# Release notes for Gluster 3.11.0 This is a major Gluster release that includes some substantial changes. The features revolve around, improvements to small file workloads, Halo replication enhancement from Facebook, some usability and performance improvements, among other bug fixes. The most notable features and changes are documented on this page. A full list of bugs that has been addressed is included further below. ## Major changes and features ### Switched to storhaug for ganesha and samba high availability *Notes for users:* High Availability (HA) support for NFS-Ganesha (NFS) and Samba (SMB) is managed by Storhaug. Like the old HA implementation, Storhaug uses Pacemaker and Corosync to manage Virtual (floating) IP addresses (VIPs) and fencing. See https://github.com/linux-ha-storage/storhaug. Storhaug packages are available in in Fedora and for several popular Linux distributions from https://download.gluster.org/pub/gluster/storhaug/ Note: Storhaug does not dictate which fencing solution should be used. There are many to choose from in most popular Linux distributions. Choose the one the best fits your environment and use it. *Limitations:* None *Known Issues:* None ### Added SELinux support for Gluster Volumes *Notes for users:* *Limitations:* *Known Issues:* ### Several memory leaks are fixed in gfapi during graph switches *Notes for users:* Gluster API (or gfapi), has had a few memory leak issues arising specifically during changes to volume graphs (volume topology or options). A few of these are addressed in this release, and more work towards ironing out the pending leaks are in the works across the next few releases. *Limitations:* None *Known Issues:* None ### get-state CLI is enhanced to provide client and brick capacity related information *Notes for users:* The get-state CLI output now optionally accommodates client related information corresponding locally running bricks as obtained from `gluster volume status |all clients`. Getting the client details is a is a relatively more costly operation and these details will only be added to the output if the get-state command is invoked with the 'detail' option. The following is the updated usage for the get-state command: ```bash # gluster get-state [] [[odir ] [file ]] [detail] ``` Other than client details, capacity related information for respective local bricks as obtained from `gluster volume status |all detail` has also been added to the get-state output. *Limitations:* Information for non-local bricks and clients connected to non-local bricks won't be available. This is a known limitation of the get-state command, since get-state command doesn't provide information on non-local bricks. *Known Issues:* ### Ability to serve negative lookups from cache has been added *Notes for users:* In this release, this feature is supported only for the SMB accesss. Before creating / renaming any file, lookups (5-6 in SMB) are sent to verify if the file already exists. By serving these lookup from the cache when possible, increases the create / rename performance by multiple folds in SMB access. Execute the following commands to enable negative-lookup cache: ```bash # gluster volume set features.cache-invalidation on # gluster volume set features.cache-invalidation-timeout 600 # gluster volume set nl-cache on ``` *Limitations:* None *Known Issues:* None ### New xlator to help developers detecting resource leaks has been added *Notes for users:* *Limitations:* *Known Issues:* ### Feature for metadata-caching/small file performance is production ready *Notes for users:* Over the course of releases several fixes and enhancements have been made to the mdcache xlator, to improve performance of small file workloads. As a result, with this release we are announcing this feature to be production ready. In order to improve the performance of directory operations of Gluster volumes, the maximum metadata (stat, xattr) caching time on the client side is increased to 10 minutes, without compromising on the consistency of the cache. Significant performance improvements can be achieved in the following workloads on FUSE and SMB access, by enabling metadata caching: - Listing of directories (recursive) - Creating files - Deleting files - Renaming files To enable metadata caching execute the following commands: ```bash # gluster volume set group metadata-cache # gluster volume set network.inode-lru-limit ``` \, is set to 50000 by default. It should be increased if the number of concurrently accessed files in the volume is very high. Increasing this number increases the memory footprint of the brick processes. *Limitations:* None *Known Issues:* None ### "Parallel Readdir" feature introduced in 3.10.0 is production ready *Notes for users:* This feature was introduced in 3.10 and was experimental in nature. Over the course of 3.10 minor releases and 3.11.0 release, this feature has been stabilized and is ready for use in production environments. For further release notes refer: [3.10.0 release notes](https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md) ### Object versioning is enabled only if bitrot is enabled *Notes for users:* *Limitations:* *Known Issues:* ### Distribute layer provides more robust transactions during directory namespace operations *Notes for users:* *Limitations:* *Known Issues:* ### gfapi extended readdirplus API has been added *Notes for users:* An extended readdirplus API `glfs_xreaddirplus` is added to get extra information along with readdirplus results on demand. This is useful for the applications (like NFS-Ganesha which needs handles) to retrieve more information along with stat in a single call, thus improving performance of work-loads involving directory listing. The API syntax and usage can be found in [`glfs.h`](https://github.com/gluster/glusterfs/blob/v3.11.0rc1/api/src/glfs.h#L810) header file. *Limitations:* This API currently has support to only return stat and handles (`glfs_object`) for each dirent of the directory, but can be extended in the future. *Known Issues:* None. ### Improved adoption of standard refcounting functions across the code *Notes for users:* This change does not impact users, it is an internal code cleanup activity that ensures that we ref count in a standard manner, thus avoiding unwanted bugs due to different implementations of the same. *Limitations:* None *Known Issues:* This standardization started with this release and is expected to continue across releases. ### Performance improvements to rebalance have been made *Notes for users:* Both crawling and migration improvement has been done in rebalance. The crawler is optimized now to split the migration load across replica and ec nodes. e.g. in case the replicating bricks are distributed over two nodes, then only one node used to do the migration. With the new optimization both the nodes divide the load among each other giving boost to migration performance. And also there have been some optimization to avoid redundant network fops. File migration now avoids syncop framework and is managed entirely by rebalance threads giving performance boost. There is a change to throttle settings in rebalance. Earlier user could set three values to rebalance which were "lazy", "normal", "aggressive", which was not flexible enough. To overcome that we have introduced number based throttle settings. User now can set numbers which is indication of the number of threads rebalance process will work with ultimately translating to the number of parallel migration. *Limitations:* *Known Issues:* ### Halo Replication feature in AFR has been introduced *Notes for users:* Halo Geo-replication is a feature which allows Gluster or NFS clients to write locally to their region (as defined by a latency "halo" or threshold if you like), and have their writes asynchronously propagate from their origin to the rest of the cluster. Clients can also write synchronously to the cluster simply by specifying a halo-latency which is very large (e.g. 10seconds) which will include all bricks. To enable halo feature execute the following commands: ```bash # gluster volume set cluster.halo-enabled yes ``` You may have to set the following following options to change defaults. `cluster.halo-shd-latency`: The threshold below which self-heal daemons will consider children (bricks) connected. `cluster.halo-nfsd-latency`: The threshold below which NFS daemons will consider children (bricks) connected. `cluster.halo-latency`: The threshold below which all other clients will consider children (bricks) connected. `cluster.halo-min-replicas`: The minimum number of replicas which are to be enforced regardless of latency specified in the above 3 options. If the number of children falls below this threshold the next best (chosen by latency) shall be swapped in. *Limitations:* None *Known Issues:* None ### FALLOCATE support with EC Notes for users: Support for FALLOCATE file operation on EC volume is added with this release. EC volumes can now support basic FALLOCATE functionality. *Limitations:* *Known Issues:* ## Bugs addressed Bugs addressed since release-3.10 are listed below. <>