glusterfs.git/events/src/utils.py, branch v3.13.2

eventsapi: HTTPS support for Webhooks

2017-10-31T06:46:56+00:00

First it tries to call URL with verify=True without specifying the cert
path, it succeeds if a webhook is HTTP or HTTPS with CA trusted
certificates(for example https://github..).

If above call fails with SSL error then it tries to get the server
certificate and calls URL again. If call fails with SSL error even after
using the certificate, then verification will be disabled and logged in
the log file.

All other errors will be catched and logged as usual.

BUG: 1506903
Change-Id: I86a3390ed48b75dffdc7848022af23a1e1d7f076
Signed-off-by: Aravinda VK

eventsapi: Add JWT signing support

2017-10-13T11:17:39+00:00

New argument added to accept secret to generate JWT token. This patch
does not affect the backward compatibility.

Usage:

    gluster-eventsapi webhook-add  [-t ] \
        [-s SECRET]

With `--token` argument, Token header will be added as is.

    Authorization: Bearer 

In case of shared secret, Gluster will generate JWT token using the
secret and then add it to Authorization header.

    Authorization: Bearer 

Secret/Token can be updated using `webhook-mod` command.

Generated token will include the following payload,

    {
       "iss": "gluster",
       "exp": EXPIRY_TIME,
       "sub": EVENT_TYPE,
       "iat": EVENT_TIME
     }

Where: iss - Issuer, exp - Expiry Time, sub - Event Type
       used as Subject, iat - Event Time used as Issue Time

BUG: 1496363
Change-Id: Ib6b6fab23fb212d7f5e9bbc9e1416a9e9813ab1b
Signed-off-by: Aravinda VK

eventsapi: Log all published events and provide option to disable logging

2016-12-08T09:22:35+00:00

Log every published event in /var/log/glusterfs/events.log, Disable logging
using,

  gluster-eventsapi config-set disable-events-log true

Also changed "log_level" config name to "log-level"

Change-Id: Ib354be0c4ca999d1ccd01b810d6cd96ebc72bcd4
BUG: 1386200
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15674
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

eventsapi: Push Messages to Webhooks in parallel

2016-12-02T14:52:55+00:00

With this patch, glustereventsd will maintain one thread per
webhook. If a webhook is slow, then all the events to that worker
will be delayed but it will not affect the other webhooks.

Note: Webhook in transit may get missed if glustereventsd reloads due to
new Webhook addition or due configuration changes.

BUG: 1357754
Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15966
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

eventsapi: Auto reload Webhooks data when modified

2016-11-17T11:10:25+00:00

glustereventsd depends on reload signal to reload the
Webhooks configurations. But if reload signal missed, no
events will be sent to newly added Webhook.

Added auto reload based on webhooks file mtime. Before pushing
events to Webhooks, reloads webhooks configurations if previously
recorded mtime is different than current mtime.

BUG: 1388862
Change-Id: I83a41d6a52d8fa1d70e88294298f4a5c396d4158
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15731
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

eventsapi: Auto convert Boolean and Int attributes

2016-10-20T06:10:44+00:00

Before publishing in JSON format, automatically convert the
attribute to "bool" or "int" if configured.

For example, instead of sending force="1", convert to bool and
send as force=True

{
    "event": "VOLUME_START",
    "name" : "gv1",
    "force": "1"
}

Convert to,

{
    "event": "VOLUME_START",
    "name" : "gv1",
    "force": true
}

BUG: 1379328
Change-Id: Iabc51fd61abc267a7c8dcf0aeac6b3c722d89649
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15574
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System

eventsapi: Add conditional import for requests library

2016-09-09T12:59:50+00:00

requests lib is used only during publishing event. gf_event python
imports utils.py, and indirectly imports requests lib even though
it is not required while sending event to eventsd.

Moved "import requests" inside the "plugin_webhook" function.

BUG: 1374587
Change-Id: Ie3c8088b43d4d7952d01352731999bf8519c73c4
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15439
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

eventsapi: Add Init scripts for different distributions

2016-09-09T10:29:33+00:00

Added init scripts for
- SysvInit(CentOS 6 or Red Hat 6)
- rc.d (FreeBSD)

Most of the latest distributions are using systemd. Support to be
added for other distributions which are not using systemd.

Removed systemctl wrapper functions(start/stop/status) from
gluster-eventsapi CLI(peer_eventsapi.py). Status and Reload
re-implemented using pid file check.

Added pid file support for glustereventsd.

Following dependencies removed
python-flask - Only used for example dashboard. User can install
if required.
python-fasteners - Not available for EPEL 6, added custom code using
fcntl as replacement.

BUG: 1365395
Change-Id: I26792eae9b11e93304f70b3997cd7d8d03b067f4
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15367
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kaleb KEITHLEY

eventsapi: Add support for Client side Events

2016-08-31T01:34:59+00:00

Client side gf_event uses ctx->cmd_args.volfile_server to push
notifications to the eventsd.

Socket server changed from Unix domain socket to UDP to support
external events.

Following to be addressed in different patch
- Port used for eventsd is 24009. Make it configurable
  Already configurable in Server side. Configurable in gf_event API
  is required.
- Auth Token yet to be added as discussed in
  https://www.gluster.org/pipermail/gluster-devel/2016-August/050324.html

Change-Id: I159acf80b681d10b82d52cfb3ffdf85cb896542d
BUG: 1367774
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15189
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

eventsapi: Gluster Eventing Feature implementation

2016-07-18T20:52:20+00:00

[Depends on http://review.gluster.org/14627]

Design is available in `glusterfs-specs`, A change from the design
is support of webhook instead of Websockets as discussed in the design

http://review.gluster.org/13115

Since Websocket support depends on REST APIs, I will add Websocket support
once REST APIs patch gets merged

Usage:
Run following command to start/stop Eventsapi server in all Peers,
which will collect the notifications from any Gluster daemon and emits
to configured client.

    gluster-eventsapi start|stop|restart|reload

Status of running services can be checked using,

    gluster-eventsapi status

Events listener is a HTTP(S) server which listens to events emited by
the Gluster. Create a HTTP Server to listen on POST and register that
URL using,

    gluster-eventsapi webhook-add  [--bearer-token ]

For example, if HTTP Server running in `http://192.168.122.188:9000`
then add that URL using,

    gluster-eventsapi webhook-add http://192.168.122.188:9000

If it expects a Token then specify it using `--bearer-token` or `-t`

We can also test Webhook if all peer nodes can send message or not
using,

    gluster-eventsapi webhook-test  [--bearer-token ]

Configurations can be viewed/updated using,

    gluster-eventsapi config-get [--name]
    gluster-eventsapi config-set  
    gluster-eventsapi config-reset 

If any one peer node was down during config-set/reset or webhook
modifications, Run sync command from good node when a peer node comes
back. Automatic update is not yet implemented.

    gluster-eventsapi sync

Basic Events Client(HTTP Server) is included with the code, Start
running the client with required port and start listening to the
events.

    /usr/share/glusterfs/scripts/eventsdash.py --port 8080

Default port is 9000, if no port is specified, once it started running
then configure gluster-eventsapi to send events to that client.

Eventsapi Client can be outside of the Cluster, it can be run event on
Windows. But only requirement is the client URL should be accessible
by all peer nodes.(Or ngrok(https://ngrok.com) like tools can be used)

Events implemented with this patch,
- Volume Create
- Volume Start
- Volume Stop
- Volume Delete
- Peer Attach
- Peer Detach

It is easy to add/support more events, since it touches Gluster cmd
code and to avoid merge conflicts I will add support for more events
once this patch merges.

BUG: 1334044
Change-Id: I316827ac9dd1443454df7deffe4f54835f7f6a08
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14248
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy