path: root/doc/release-notes/3.11.0.md

# Release notes for Gluster 3.11.0

This is a major Gluster release that includes some substantial changes. The
features revolve around, improvements to small file workloads, Halo replication
enhancement from Facebook, some usability and performance improvements, among
other bug fixes.

The most notable features and changes are documented on this page. A full list
of bugs that has been addressed is included further below.

## Major changes and features

### Switched to storhaug for ganesha and samba high availability
*Notes for users:*

High Availability (HA) support for NFS-Ganesha (NFS) and Samba (SMB)
is managed by Storhaug. Like the old HA implementation, Storhaug uses
Pacemaker and Corosync to manage Virtual (floating) IP addresses (VIPs)
and fencing. See https://github.com/linux-ha-storage/storhaug.

Storhaug packages are available in in Fedora and for several popular
Linux distributions from https://download.gluster.org/pub/gluster/storhaug/ 

Note: Storhaug does not dictate which fencing solution should be used.
There are many to choose from in most popular Linux distributions.
Choose the one the best fits your environment and use it.

*Limitations:*

None

*Known Issues:*

None

### Added SELinux support for Gluster Volumes
*Notes for users:*

*Limitations:*

*Known Issues:*

### Several memory leaks are fixed in gfapi during graph switches
*Notes for users:*

Gluster API (or gfapi), has had a few memory leak issues arising specifically
during changes to volume graphs (volume topology or options). A few of these are
addressed in this release, and more work towards ironing out the pending leaks
are in the works across the next few releases.

*Limitations:*
None

*Known Issues:*
None

### get-state CLI is enhanced to provide client and brick capacity related information
*Notes for users:*

The get-state CLI output now optionally accommodates client related information
corresponding locally running bricks as obtained from
`gluster volume status <volname>|all clients`. Getting the client details is a
is a relatively more costly operation and these details will only be added to
the output if the get-state command is invoked with the 'detail' option. The
following is the updated usage for the get-state command:

```bash
 # gluster get-state [<daemon>] [[odir </path/to/output/dir/>] [file <filename>]] [detail]
```

Other than client details, capacity related information for respective local
bricks as obtained from `gluster volume status <volname>|all detail` has also
been added to the get-state output.

*Limitations:*

Information for non-local bricks and clients connected to non-local bricks
won't be available. This is a known limitation of the get-state command, since
get-state command doesn't provide information on non-local bricks.

*Known Issues:*

### Ability to serve negative lookups from cache has been added
*Notes for users:*
In this release, this feature is supported only for the SMB accesss.
Before creating / renaming any file, lookups (5-6 in SMB) are sent
to verify if the file already exists. By serving these lookup from the
cache when possible, increases the create / rename performance by multiple
folds in SMB access.
Execute the following commands to enable negative-lookup cache: 
```bash
# gluster volume set <volname> features.cache-invalidation on
# gluster volume set <volname> features.cache-invalidation-timeout 600
# gluster volume set <VOLNAME> nl-cache on
```

*Limitations:*
None

*Known Issues:*
None

### New xlator to help developers detecting resource leaks has been added
*Notes for users:*

*Limitations:*

*Known Issues:*

### Feature for metadata-caching/small file performance is production ready
*Notes for users:*

Over the course of releases several fixes and enhancements have been made to
the mdcache xlator, to improve performance of small file workloads. As a
result, with this release we are announcing this feature to be production ready.

In order to improve the performance of directory operations of Gluster volumes,
the maximum metadata (stat, xattr) caching time on the client side is increased
to 10 minutes, without compromising on the consistency of the cache. Significant
performance improvements can be achieved in the following workloads on FUSE and
SMB access, by enabling metadata caching:

- Listing of directories (recursive)
- Creating files
- Deleting files
- Renaming files

To enable metadata caching execute the following commands:
```bash
# gluster volume set group metadata-cache
# gluster volume set network.inode-lru-limit <n>
```
\<n\>, is set to 50000 by default. It should be increased if the number of
concurrently accessed files in the volume is very high. Increasing this number
increases the memory footprint of the brick processes.

*Limitations:*
None

*Known Issues:*
None

### "Parallel Readdir" feature introduced in 3.10.0 is production ready
*Notes for users:*

This feature was introduced in 3.10 and was experimental in nature. Over the
course of 3.10 minor releases and 3.11.0 release, this feature has been
stabilized and is ready for use in production environments.

For further release notes refer: [3.10.0 release notes](https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md)

### Object versioning is enabled only if bitrot is enabled
*Notes for users:*

*Limitations:*

*Known Issues:*

### Distribute layer provides more robust transactions during directory namespace operations
*Notes for users:*

*Limitations:*

*Known Issues:*

### gfapi extended readdirplus API has been added
*Notes for users:*

An extended readdirplus API `glfs_xreaddirplus` is added to get extra information along
 with readdirplus results on demand. This is useful for the applications (like NFS-Ganesha
 which needs handles) to retrieve more information along with stat in a single call, thus
 improving performance of work-loads involving directory listing.

The API syntax and usage can be found in
 [`glfs.h`](https://github.com/gluster/glusterfs/blob/v3.11.0rc1/api/src/glfs.h#L810)
 header file.

*Limitations:*

This API currently has support to only return stat and handles (`glfs_object`) for each dirent
 of the directory, but can be extended in the future.

*Known Issues:*

None.

### Improved adoption of standard refcounting functions across the code
*Notes for users:*

This change does not impact users, it is an internal code cleanup activity
that ensures that we ref count in a standard manner, thus avoiding unwanted
bugs due to different implementations of the same.

*Limitations:*
None

*Known Issues:*
This standardization started with this release and is expected to continue
across releases.

### Performance improvements to rebalance have been made
*Notes for users:*
Both crawling and migration improvement has been done in rebalance.
The crawler is optimized now to split the migration load across replica and ec nodes.
e.g. in case the replicating bricks are distributed over two nodes, then only one node used
to do the migration. With the new optimization both the nodes divide the load among each
other giving boost to migration performance. And also there have been some optimization
to avoid redundant network fops.

File migration now avoids syncop framework and is  managed entirely by rebalance threads giving
performance boost.

There is a change to throttle settings in rebalance. Earlier user could set three values to
rebalance which were "lazy", "normal", "aggressive", which was not flexible enough. To overcome
that we have introduced number based throttle settings. User now can set numbers which is indication
of the number of threads rebalance process will work with ultimately translating to the number of
parallel migration.

*Limitations:*

*Known Issues:*

### Halo Replication feature in AFR has been introduced
*Notes for users:*
Halo Geo-replication is a feature which allows Gluster or NFS clients to write
locally to their region (as defined by a latency "halo" or threshold if you
like), and have their writes asynchronously propagate from their origin to the
rest of the cluster.  Clients can also write synchronously to the cluster
simply by specifying a halo-latency which is very large (e.g. 10seconds) which
will include all bricks.
To enable halo feature execute the following commands:
```bash
# gluster volume set cluster.halo-enabled yes
```

You may have to set the following following options to change defaults.
`cluster.halo-shd-latency`:  The threshold below which self-heal daemons will
consider children (bricks) connected.

`cluster.halo-nfsd-latency`: The threshold below which NFS daemons will consider
children (bricks) connected.

`cluster.halo-latency`: The threshold below which all other clients will
consider children (bricks) connected.

`cluster.halo-min-replicas`: The minimum number of replicas which are to
be enforced regardless of latency specified in the above 3 options.
If the number of children falls below this threshold the next
best (chosen by latency) shall be swapped in.


*Limitations:*
None

*Known Issues:*
None

### FALLOCATE support with EC
Notes for users:

Support for FALLOCATE file operation on EC volume is added with this release.
EC volumes can now support basic FALLOCATE functionality.

*Limitations:*

*Known Issues:*

## Bugs addressed

Bugs addressed since release-3.10 are listed below.

<<TO DO>>