6 14:50:47 GMT Expires: Thu, 07 Feb 2036 14:50:47 GMT ETag: "c744957ba85e78d4fa9a99d06fb44fda9e6083f1" # Release notes for Gluster 4.0.0 The Gluster community celebrates 13 years of development with this latest release, Gluster 4.0. This release enables improved integration with containers, an enhanced user experience, and a next-generation management framework. The 4.0 release helps cloud-native app developers choose Gluster as the default scale-out distributed file system. A selection of the important features and changes are documented on this page. A full list of bugs that have been addressed is included further below. - [Announcements](#announcements) - [Major changes and features](#major-changes-and-features) - [Major issues](#major-issues) - [Bugs addressed in the release](#bugs-addressed) ## Announcements 1. As 3.13 was a short term maintenance release, features which have been included in that release are available with 4.0.0 as well.These features may be of interest to users upgrading to 4.0.0 from older than 3.13 releases. The 3.13 [release notes](http://docs.gluster.org/en/latest/release-notes/) captures the list of features that were introduced with 3.13. **NOTE:** As 3.13 was a short term maintenance release, it will reach end of life (EOL) with the release of 4.0.0. ([reference](https://www.gluster.org/release-schedule/)) 2. Releases that receive maintenance updates post 4.0 release are, 3.10, 3.12, 4.0 ([reference](https://www.gluster.org/release-schedule/)) 3. With this release, the CentOS storage SIG will not build server packages for CentOS6. Server packages will be available for CentOS7 only. For ease of migrations, client packages on CentOS6 will be published and maintained. **NOTE**: This change was announced [here](http://lists.gluster.org/pipermail/gluster-users/2018-January/033212.html) ## Major changes and features Features are categorized into the following sections, - [Management](#management) - [Monitoring](#monitoring) - [Performance](#performance) - [Geo-replication](#geo-replication) - [Standalone](#standalone) - [Developer related](#developer-related) ### Management GlusterD2 (GD2) is new management daemon for Gluster-4.0. It is a complete rewrite, with all new internal core frameworks, that make it more scalable, easier to integrate with and has lower maintenance requirements. A [quick start guide](https://github.com/gluster/glusterd2/blob/master/doc/quick-start-user-guide.md) is available to get started with GD2. GD2 in Gluster-4.0 is a technical preview release. It is not recommended for production use. For the current release glusterd is the preferred management daemon. More information is available in the [Limitations](#limitations) section. GD2 brings many new changes and improvements, that affect both users and developers. #### Features The most significant new features brought by GD2 are below. ##### Native REST APIs GD2 exposes all of its management functionality via [ReST APIs](https://github.com/gluster/glusterd2/blob/master/doc/endpoints.md). The ReST APIs accept and return data encoded in JSON. This enables external projects such as [Heketi](https://github.com/heketi/heketi) to be better integrated with GD2. ##### CLI GD2 provides a new CLI, `glustercli`, built on top of the ReST API. The CLI retains much of the syntax of the old `gluster` command. In addition we have, - Improved CLI help messages - Auto completion for sub commands - Improved CLI error messages on failure - Framework to run `glustercli` from outside the Cluster. In this release, the following CLI commands are available, - Peer management - Peer Probe/Attach - Peer Detach - Peer Status - Volume Management - Create/Start/Stop/Delete - Expand - Options Set/Get - Bitrot - Enable/Disable - Configure - Status - Geo-replication - Create/Start/Pause/Resume/Stop/Delete - Configure - Status ##### Configuration store GD2 uses [etcd](https://github.com/coreos/etcd/) to store the Gluster pool configuration, which solves the config synchronize issues reported against the Gluster management daemon. GD2 embeds etcd, and automatically creates and manages an etcd cluster when forming the trusted storage pool. If required, GD2 can also connect to an already existing etcd cluster. ##### Transaction Framework GD2 brings a newer more flexible distributed framework, to help it perform actions across the storage pool. The transaction framework provides better control for choosing peers for a Gluster operation and it also provides a mechanism to roll back the changes when something goes bad. ##### Volume Options GD2 intelligently fetches and builds the list of volume options by directly reading `xlators` `*.so` files. It does required validations during volume set without maintaining duplicate list of options. This avoids lot of issues which can happen due to mismatch in the information between Glusterd and xlator shared libraries. Volume options listing is also improved, to clearly distinguish configured options and default options. Work is still in progress to categorize these options and tune the list for better understanding and ease of use. ##### Volfiles generation and management GD2 has a newer and better structured way for developers to define volfile structure. The new method reduces the effort required to extend graphs or add new graphs. Also, volfiles are generated in single peer and stored in `etcd` store. This is very important for scalability since Volfiles are not stored in every node. ##### Security GD2 supports TLS for ReST and internal communication, and authentication for the ReST API.If enabled, ReST APIs are currently limited to CLI, or the users who have access to the Token file present in `$GLUSTERD2_WORKDIR/auth` file. ##### Features integration - Self Heal Self Heal feature integrated for the new Volumes created using Glusterd2. ##### Geo-replication With GD2 integration Geo-replication setup becomes very easy. If Master and Remote volume are available and running, Geo-replication can be setup with just a single command. ``` glustercli geo-replication create :: ``` Geo-replication status is improved, Status clearly distinguishes the multiple session details in status output. Order of status rows was not predictable in earlier releases. It was very difficult to correlate the Geo-replication status with Bricks. With this release, Master worker status rows will always match with Bricks list in Volume info. Status can be checked using, ``` glustercli geo-replication status glustercli geo-replication status :: ``` All the other commands are available as usual. Limitations: - On Remote nodes, Geo-replication is not yet creates the log directories. As a workaround, create the required log directories in Remote Volume nodes. ##### Events APIs Events API feature is integrated with GD2. Webhooks can be registered to listen for GlusterFS events. Work is in progress for exposing an REST API to view all the events happened in last 15 minutes. #### Limitations ##### Backward compatibility GD2 is not backwards compatible with the older GlusterD. Heterogeneous clusters running both GD2 and GlusterD are not possible. GD2 retains compatibility with Gluster-3.x clients. Old clients will still be able to mount and use volumes exported using GD2. ##### Upgrade and migration GD2 does not support upgrade from Gluster-3.x releases, in Gluster-4.0. Gluster-4.0 will be shipping with both GD2 and the existing GlusterD. Users will be able to upgrade to Gluster-4.0 while continuing to use GlusterD. In Gluster-4.1, users will be able to migrate from GlusterD to GD2. Further, upgrades from Gluster-4.1 running GD2 to higher Gluster versions would be supported from release 4.1 onwards. Post Gluster-4.1, GlusterD would be maintained for a couple of releases, post which the only option to manage the cluster would be GD2. ##### Missing and partial commands Not all commands from GlusterD, have been implemented for GD2. Some have been only partially implemented. This means not all GlusterFS features are available in GD2. We aim to bring most of the commands back in Gluster-4.1. ##### Recovery from full shutdown With GD2, the process of recovery from a situation of a full cluster shutdown requires reading the [document available](https://github.com/gluster/glusterd2/wiki/Recovery) as well as some expertise. #### Known Issues ##### 2-node clusters GD2 does not work well in 2-node clusters. Two main issues exist in this regard. - Restarting GD2 fails in 2-node clusters [#352](https://github.com/gluster/glusterd2/issues/352) - Detach fails in 2-node clusters [#332](https://github.com/gluster/glusterd2/issues/332) So it is recommended right now to run GD2 only in clusters of 3 or larger. ##### Other issues Other known issues are tracked on [github issues](https://github.com/gluster/glusterd2/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+) right now. Please file any other issue you find on github issues. ### Monitoring Till date, the absence of support for live monitoring on GlusterFS created constrained user experience for both users and developers. [Statedump](docs.gluster.org/en/latest/Troubleshooting/statedump/) is useful for debugging, but is heavy for live monitoring. Further, the existence of `debug/io-stats` translator was not known to many and `gluster volume profile` was not recommended as it impacted performance. In this release, GlusterFS enables a lightweight method to access internal information and avoids the performance penalty and complexities of previous approaches. #### 1. Metrics collection across every FOP in every xlator **Notes for users:** Now, Gluster now has in-built latency measures in the xlator abstraction, thus enabling capture of metrics and usage patterns across workloads. These measures are currently enabled by default. **Limitations:** This feature is auto-enabled and cannot be disabled. #### 2. Monitoring support **Notes for users:** Currently, the only project which consumes metrics and provides basic monitoring is [glustermetrics](https://github.com/amarts/glustermetrics), which provides a good idea on how to utilize the metrics dumped from the processes. Users can send SIGUSR2 signal to the process to dump the metrics, in `/var/run/gluster/metrics/` directory. **Limitations:** Currently core gluster stack and memory management systems provide metrics. A framework to generate more metrics is present for other translators and core components. However, additional metrics are not added in this release. ### Performance #### 1. EC: Make metadata [F]GETXATTR operations faster **Notes for users:** Disperse translator has made performance improvements to the [F]GETXATTR operation. Workloads involving heavy use of extended attributes on files and directories, will gain from the improvements made. #### 2. Allow md-cache to serve nameless lookup from cache **Notes for users:** The md-cache translator is enhanced to cache nameless lookups (typically seen with NFS workloads). This helps speed up overall operations on the volume reducing the number of lookups done over the network. Typical workloads that will benefit from this enhancement are, - NFS based access - Directory listing with FUSE, when ACLs are enabled #### 3. md-cache: Allow runtime addition of xattrs to the list of xattrs that md-cache caches **Notes for users:** md-cache was enhanced to cache extended attributes of a file or directory, for gluster specific attributes. This has now been enhanced to cache user provided attributes (xattrs) as well. To add specific xattrs to the cache list, use the following command: ``` # gluster volume set xattr-cache-list ",,..." ``` Existing options, such as "cache-samba-metadata" "cache-swift-metadata" continue to function. The new option