summaryrefslogtreecommitdiffstats
path: root/doc/admin-guide/en-US/admin_geo-replication.xml
blob: 4691116acb82dd6c407ad71036fb34a28c6f6fbc (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "docbookV4.5/docbookx.dtd" []>
<chapter id="chap-Administration_Guide-Geo_Rep">
  <title>Managing Geo-replication</title>
  <para>Geo-replication provides a continuous, asynchronous, and incremental replication service from one site to another over Local Area Networks (LANs), Wide Area Network (WANs), and across the Internet. </para>
  <para>Geo-replication uses a master–slave model, whereby replication and mirroring occurs between the following partners:</para>
  <itemizedlist>
    <listitem>
      <para>Master – a GlusterFS volume </para>
    </listitem>
    <listitem>
      <para>Slave – a slave which can be of the following types: </para>
      <itemizedlist>
        <listitem>
          <para>A local directory which can be represented as file URL like <filename>file:///path/to/dir</filename>. You can use shortened form, for example, <filename> /path/to/dir</filename>.</para>
        </listitem>
        <listitem>
          <para>A GlusterFS Volume - Slave volume can be either a local volume like <filename>gluster://localhost:volname</filename> (shortened form - <filename>:volname</filename>) or a volume served by different host like <filename>gluster://host:volname</filename> (shortened form - <filename>host:volname</filename>).</para>
        </listitem>
      </itemizedlist>
      <note>
        <para> Both of the above types can be accessed remotely using SSH tunnel. To use SSH, add an SSH prefix to either a file URL or gluster type URL. For example, <literal> ssh://root@remote-host:/path/to/dir</literal> (shortened form - <literal>root@remote-host:/path/to/dir</literal>) or <literal>ssh://root@remote-host:gluster://localhost:volname</literal> (shortened from - <literal>root@remote-host::volname</literal>). </para>
      </note>
    </listitem>
  </itemizedlist>
  <para>This section introduces  Geo-replication, illustrates the various deployment scenarios, and explains how to configure the system to provide replication and mirroring in your environment. </para>
  <section id="chap-Administration_Guide-Geo_Rep-Replicated_volumes">
    <title>Replicated Volumes vs Geo-replication</title>
    <para>The following table lists the difference between replicated volumes and geo-replication:</para>
    <informaltable frame="all">
      <tgroup cols="2">
        <colspec colname="c1"/>
        <colspec colname="c2"/>
        <thead>
          <row>
            <entry>Replicated Volumes</entry>
            <entry>Geo-replication</entry>
          </row>
        </thead>
        <tbody>
          <row>
            <entry>Mirrors data across nodes in a cluster</entry>
            <entry>Mirrors data across geographically distributed clusters </entry>
          </row>
          <row>
            <entry>Provides high-availability</entry>
            <entry>Ensures backing up of data for disaster recovery</entry>
          </row>
          <row>
            <entry>Synchronous replication (each and every file modify operation is sent across all the bricks)</entry>
            <entry>Asynchronous replication (checks for the changes in files periodically and syncs them on detecting differences) </entry>
          </row>
        </tbody>
      </tgroup>
    </informaltable>
  </section>
  <section id="chap-Administration_Guide-Geo_Rep-Preparation">
    <title>Preparing to Deploy Geo-replication</title>
    <para>This section provides an overview of the  Geo-replication deployment scenarios, describes how you can check the minimum system requirements, and explores common deployment scenarios.</para>
    <itemizedlist>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Deployment_options"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Deployment_Overview"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Minimum_Reqs"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Settingup_Environment"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Settingup_Slave"/></para>
      </listitem>
    </itemizedlist>
    <section id="chap-Administration_Guide-Geo_Rep-Preparation-Deployment_options">
      <title>Exploring Geo-replication Deployment Scenarios</title>
      <para>Geo-replication provides an incremental replication service over Local Area Networks (LANs), Wide Area Network (WANs), and across the Internet. This section illustrates the most common deployment scenarios for  Geo-replication, including the following: </para>
      <itemizedlist>
        <listitem>
          <para>Geo-replication over LAN
</para>
        </listitem>
        <listitem>
          <para>Geo-replication over WAN
</para>
        </listitem>
        <listitem>
          <para>Geo-replication over the Internet</para>
        </listitem>
        <listitem>
          <para>Multi-site cascading Geo-replication</para>
        </listitem>
      </itemizedlist>
      <para><emphasis role="bold">Geo-replication over LAN</emphasis></para>
      <para>You can configure  Geo-replication to mirror data over a Local Area Network. </para>
      <mediaobject>
        <textobject>
          <phrase>Geo-replication over LAN</phrase>
        </textobject>
        <imageobject>
          <imagedata fileref="images/Geo-Rep_LAN.png"/>
        </imageobject>
      </mediaobject>
      <para><emphasis role="bold">Geo-replication over WAN</emphasis></para>
      <para>You can configure Geo-replication to replicate data over a Wide Area Network.</para>
      <mediaobject>
        <textobject>
          <phrase>
            <phrase>Geo-replication over WAN</phrase>
          </phrase>
        </textobject>
        <imageobject>
          <imagedata fileref="images/Geo-Rep_WAN.png"/>
        </imageobject>
      </mediaobject>
      <para><emphasis role="bold">Geo-replication over Internet</emphasis></para>
      <para>You can configure Geo-replication to mirror data over the Internet.</para>
      <mediaobject>
        <textobject>
          <phrase>
            <phrase>Geo-replication over Internet</phrase>
          </phrase>
        </textobject>
        <imageobject>
          <imagedata fileref="images/Geo-Rep03_Internet.png"/>
        </imageobject>
      </mediaobject>
      <para><emphasis role="bold">Multi-site cascading Geo-replication</emphasis> </para>
      <para>You can configure Geo-replication to mirror data in a cascading fashion across multiple sites. </para>
      <mediaobject>
        <textobject>
          <phrase>
            <phrase>Multi-site cascading Geo-replication </phrase>
          </phrase>
        </textobject>
        <imageobject>
          <imagedata fileref="images/Geo-Rep04_Cascading.png"/>
        </imageobject>
      </mediaobject>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Preparation-Deployment_Overview">
      <title>Geo-replication Deployment Overview</title>
      <para>Deploying Geo-replication involves the following steps:</para>
      <orderedlist>
        <listitem>
          <para>Verify that your environment matches the minimum system requirements. For more information, see <xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Minimum_Reqs"/>.</para>
        </listitem>
        <listitem>
          <para>Determine the appropriate deployment scenario. For more information, see <xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Deployment_options"/>.</para>
        </listitem>
        <listitem>
          <para>Start Geo-replication on master and slave systems, as required. For more information, see <xref linkend="chap-Administration_Guide-Geo_Rep-Starting"/>.</para>
        </listitem>
      </orderedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Preparation-Minimum_Reqs">
      <title>Checking Geo-replication Minimum Requirements</title>
      <para condition="gfs">Before deploying GlusterFS Geo-replication, verify that your systems match the minimum requirements. </para>
      <para condition="gfs">The following table outlines the minimum requirements for both master and slave nodes within your environment:</para>
      <informaltable frame="all" condition="gfs">
        <tgroup cols="3">
          <colspec colname="c1"/>
          <colspec colname="c2"/>
          <colspec colname="c3"/>
          <thead>
            <row>
              <entry>Component</entry>
              <entry>Master</entry>
              <entry>Slave</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>Operating System </entry>
              <entry>GNU/Linux</entry>
              <entry>GNU/Linux</entry>
            </row>
            <row>
              <entry>Filesystem</entry>
              <entry>GlusterFS 3.2 or higher</entry>
              <entry>GlusterFS 3.2 or higher, ext3, ext4, or XFS (any other POSIX compliant file system would work, but has not been tested extensively) </entry>
            </row>
            <row>
              <entry>Python </entry>
              <entry>Python 2.4 (with ctypes external module), or Python 2.5 (or higher)</entry>
              <entry>Python 2.4 (with ctypes external module), or Python 2.5 (or higher)</entry>
            </row>
            <row>
              <entry>Secure shell </entry>
              <entry>OpenSSH version 4.0 (or higher)</entry>
              <entry>SSH2-compliant daemon </entry>
            </row>
            <row>
              <entry>Remote synchronization</entry>
              <entry>rsync 3.0.0 or higher </entry>
              <entry>rsync 3.0.0 or higher </entry>
            </row>
            <row>
              <entry>FUSE </entry>
              <entry>GlusterFS supported versions </entry>
              <entry>GlusterFS supported versions </entry>
            </row>
          </tbody>
        </tgroup>
      </informaltable>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Preparation-Settingup_Environment">
      <title>Setting Up the Environment for Geo-replication</title>
      <para><emphasis role="bold">Time Synchronization</emphasis> </para>
      <itemizedlist>
        <listitem>
          <para>All servers that are part of a geo-replication master volume need to have their clocks in sync. You are recommended to set up NTP (Network Time Protocol) daemon service to keep the clocks in sync.</para>
          <para>For example:  In a Replicated volume where brick1 of the master is at 12.20 hrs and brick 2 of the master is at 12.10 hrs with 10 minutes time lag, all the changes in brick2 between this period may go unnoticed during synchronization of files with Slave.</para>
          <para>For more information on setting up NTP daemon, see <ulink url="http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Migration_Planning_Guide/ch04s07.html"/>.</para>
        </listitem>
      </itemizedlist>
      <para><emphasis role="bold">To setup Geo-replication for SSH </emphasis></para>
      <para>Password-less login has to be set up between the host machine (where geo-replication start command will be issued) and  the remote machine (where slave process should be launched through SSH).</para>
      <orderedlist>
        <listitem>
          <para>On the node where geo-replication start commands are to be issued, run the following command:</para>
          <para><command># ssh-keygen -f /var/lib/glusterd/geo-replication/secret.pem</command>
</para>
          <para>Press Enter twice to avoid passphrase.
</para>
        </listitem>
        <listitem>
          <para>Run the following command on master for all the slave hosts: </para>
          <para><command># ssh-copy-id -i /var/lib/glusterd/geo-replication/secret.pem.pub <varname>user</varname>@<varname>slavehost</varname></command></para>
        </listitem>
      </orderedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Preparation-Settingup_Slave">
      <title>Setting Up the Environment for a Secure Geo-replication Slave</title>
      <para>You can configure a secure slave using SSH so that master is granted
restricted access. With GlusterFS 3.3, you need not specify slave
configuration parameters on the master-side. For example, the master does not require the location of
the rsync program on slave but the slave must ensure that rsync is in
the PATH of the user which the master connects using SSH. The only
information that master and slave have to negotiate are the slave-side
user account, slave&apos;s resources and
the master&apos;s public key. Secure access to the slave can be established
using the following options:</para>
      <itemizedlist>
        <listitem>
          <para>Restricting Remote Command Execution</para>
        </listitem>
        <listitem>
          <para>Using <filename>Mountbroker</filename> for Slaves</para>
        </listitem>
        <listitem>
          <para>Using IP based Access Control</para>
        </listitem>
      </itemizedlist>
      <para><emphasis role="bold">Backward Compatibility</emphasis> </para>
      <para>Your existing Geo-replication environment will work with GlusterFS
              3.3, except for the following:</para>
      <itemizedlist>
        <listitem>
          <para>The process of secure reconfiguration affects only the GlusterFS
instance on slave. The changes are transparent to master with the
exception that you may have to  change the SSH target to an unprivileged
 account on the slave.</para>
        </listitem>
        <listitem>
          <para>The following are some exceptions where backward compatibility cannot be provided:</para>
          <para><itemizedlist>
              <listitem>
                <para>Geo-replication URLs which specify the slave resource include the following special characters: space, *, ?, [;</para>
              </listitem>
              <listitem>
                <para>Slave does not have glusterd running.</para>
              </listitem>
            </itemizedlist></para>
        </listitem>
      </itemizedlist>
      <section>
        <title>Restricting Remote Command Execution</title>
        <para>If you restrict remote command execution, then the slave audits commands
coming from the master and only the pre-configured commands are allowed. The slave also provides access only
to the files which are pre-configured to be read or manipulated by the master.</para>
        <para>To restrict remote command execution:</para>
        <orderedlist>
          <listitem>
            <para>Identify the location of the gsyncd helper utility on Slave. This utility is installed in <filename>PREFIX/libexec/glusterfs/gsyncd</filename>, where PREFIX is a compile-time parameter of glusterfs. For example,  <filename>--prefix=PREFIX</filename> to the configure script with the following common  values<filename> /usr, /usr/local, and /opt/glusterfs/glusterfs_version</filename>.</para>
          </listitem>
          <listitem>
            <para>Ensure that command invoked from master to slave is passed through the slave&apos;s gsyncd utility. </para>
            <para>You can use either of the following two options:</para>
            <itemizedlist>
              <listitem>
                <para>Set gsyncd with an absolute path as the shell for the account
which the master connects through SSH. If you need to use a privileged
account, then set it up by creating  a new user with UID 0. </para>
              </listitem>
              <listitem>
                <para>Setup key authentication with command enforcement to gsyncd. You must prefix the copy of master&apos;s public key in the Slave account&apos;s <filename>authorized_keys</filename> file with the following command:</para>
                <para><filename>command=&lt;path to gsyncd&gt;</filename>. </para>
                <para>For example, <command>command=&quot;PREFIX/glusterfs/gsyncd&quot; ssh-rsa AAAAB3Nza....</command></para>
              </listitem>
            </itemizedlist>
          </listitem>
        </orderedlist>
      </section>
      <section>
        <title>Using Mountbroker for Slaves </title>
        <para><filename>mountbroker</filename> is a new service of glusterd. This service allows an
unprivileged process to own a GlusterFS mount. This is accomplished by registering a label
(and DSL (Domain-specific language) options ) with glusterd through the
glusterd volfile. Using CLI, you can send a mount request to glusterd and
receive an alias (symlink) to the mounted volume.</para>
        <para>The unprivileged process/agent uses the
mountbroker service of glusterd to set up an auxiliary gluster mount. The mount
is setup so as to allow only that agent to provide administrative
level access to the particular volume.</para>
        <para><emphasis role="bold">To setup an auxiliary gluster mount for the agent</emphasis>:</para>
        <orderedlist>
          <listitem>
            <para>Create a new group. For example, <filename>geogroup</filename>.</para>
          </listitem>
          <listitem>
            <para>Create a unprivileged account. For example, <filename> geoaccount</filename>. Make it a member of  <filename> geogroup</filename>.</para>
          </listitem>
          <listitem>
                  <para>Create a new directory as superuser to be used as mountbroker's root. </para>
          </listitem>
          <listitem>
                  <para> Change the  permission of the directory to <emphasis role="italic">0711.</emphasis> </para>
          </listitem>
          <listitem>
            <para>Add the following options to the glusterd volfile, located at /etc/glusterfs/glusterd.vol, assuming the name of the slave gluster volume as <filename>slavevol</filename>:</para>
            <para><command>option mountbroker-root /var/mountbroker-root </command></para>
            <para><command>option mountbroker-geo-replication.geoaccount slavevol</command></para>
            <para><command>option geo-replication-log-group geogroup</command></para>
           <para>A sample glusterd volfile along with default options:</para>
            <para><screen>volume management
    type mgmt/glusterd
    option working-directory /etc/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off

    option mountbroker-root /var/mountbroker-root
    option mountbroker-geo-replication.geoaccount slavevol
    option geo-replication-log-group geogroup
end-volume</screen></para>
            <para>If you host multiple slave volumes, you can repeat step 2. for each of the slave volumes and add the following options to the <filename>volfile</filename>:</para>
            <para><screen>option mountbroker-geo-replication.geoaccount2 slavevol2
option mountbroker-geo-replication.geoaccount3 slavevol3</screen></para>
          </listitem>
          <listitem>
            <para>Setup Master to access Slave as <filename>geoaccount@Slave</filename>.</para>
            <para>You can add multiple slave volumes within the same account (geoaccount) by providing comma-separated list of slave
                    volumes (without spaces) as the argument of <command>mountbroker-geo-replication.geogroup</command>. You can also have multiple options of the form <command>mountbroker-geo-replication.*</command>. It is recommended to  use one service account per Master machine. For example, if there are multiple slave volumes on Slave for the master machines Master1, Master2, and Master3, then create a dedicated service user on Slave for them by repeating Step 2. for each (like geogroup1, geogroup2, and geogroup3), and then add the following corresponding options to the volfile:
</para>
            <para><command>option mountbroker-geo-replication.geoaccount1 slavevol11,slavevol12,slavevol13</command></para>
            <para><command>option mountbroker-geo-replication.geoaccount2 slavevol21,slavevol22</command></para>
            <para><command>option mountbroker-geo-replication.geoaccount3 slavevol31</command></para>
            <para>
Now set up Master1 to ssh to geoaccount1@Slave, etc.
</para>
            <para>You must restart glusterd to make the configuration changes effective. </para>
          </listitem>
        </orderedlist>
      </section>
      <section>
        <title>Using IP based Access Control</title>
        <para>You can provide access control for the slave resources using IP
                addresses. You can use method for both Gluster volume and and
                file tree slaves, but in this section, we are focusing on file tree slaves.</para>
        <para>To set IP address based access control for file tree slaves:</para>
        <orderedlist>
          <listitem>
            <para>Set a general restriction for accessibility of file tree resources:
</para>
            <para><command># gluster volume geo-replication &apos;/*&apos; config allow-network ::1,127.0.0.1 </command></para>
            <para>This will refuse all requests for spawning slave agents except for
requests initiated locally.</para>
          </listitem>
          <listitem>
            <para>If you want the to lease file tree at <filename>/data/slave-tree</filename> to Master, enter the following command:</para>
            <para><command># gluster volume geo-replication<varname> /data/slave-tree </varname>config allow-network <varname>MasterIP</varname></command></para>
            <para><varname>MasterIP</varname> is the IP address of Master. The slave agent spawn request from
master will be accepted if it is executed at  <filename>/data/slave-tree</filename>.</para>
          </listitem>
        </orderedlist>
        <para>If the Master side network configuration does not enable the Slave to
recognize the exact IP address of Master, you can use CIDR notation to
specify a subnet instead of a single IP address as MasterIP or even
comma-separated lists of CIDR subnets.</para>
        <para>If you want to extend IP based access control to gluster slaves, use the following command:</para>
        <para><command># gluster volume geo-replication &apos;*&apos; config allow-network ::1,127.0.0.1</command></para>
      </section>
    </section>
  </section>
  <section id="chap-Administration_Guide-Geo_Rep-Starting">
    <title>Starting Geo-replication</title>
    <para>This section describes how to configure and start Gluster Geo-replication in your storage environment, and verify that it is functioning correctly. </para>
    <itemizedlist>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Start"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Verify"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Display"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Configure"/></para>
      </listitem>
      <listitem>
        <para><xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Stop"/></para>
      </listitem>
    </itemizedlist>
    <section id="chap-Administration_Guide-Geo_Rep-Starting-Start">
      <title>Starting Geo-replication</title>
      <para>To start Gluster Geo-replication </para>
      <itemizedlist>
        <listitem>
          <para>Start geo-replication between the hosts using the following command:
 </para>
          <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> start</command>
</para>
          <para>For example:
</para>
          <para><programlisting># gluster volume geo-replication Volume1 example.com:/data/remote_dir start
Starting geo-replication session between Volume1
example.com:/data/remote_dir has been successful</programlisting></para>
          <para><note>
              <para>You may need to configure the Geo-replication service before
                      starting it. For more information, see <xref linkend="chap-Administration_Guide-Geo_Rep-Starting-Configure"/>.</para>
            </note></para>
        </listitem>
      </itemizedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Starting-Verify">
      <title>Verifying Successful Deployment</title>
      <para>You can use the gluster command to verify the status of Gluster Geo-replication in your environment.</para>
      <para><emphasis role="bold">To verify the status Gluster Geo-replication</emphasis></para>
      <itemizedlist>
        <listitem>
          <para>Verify the status by issuing the following command on host:</para>
          <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> status</command>
</para>
          <para>For example: 
</para>
          <para><command># gluster volume geo-replication Volume1 example.com:/data/remote_dir status</command> 
</para>
          <para><programlisting># gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 root@example.com:/data/remote_dir  Starting....</programlisting> 
</para>
        </listitem>
      </itemizedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Starting-Display">
      <title>Displaying Geo-replication Status Information</title>
      <para>You can display status information about a specific geo-replication master session, or a particular master-slave session, or all geo-replication sessions, as needed.</para>
      <para><emphasis role="bold">To display geo-replication status information</emphasis></para>
      <itemizedlist>
        <listitem>
          <para>Display information of all geo-replication sessions using the following command:</para>
          <para><programlisting># gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 root@example.com:/data/remote_dir  Starting....</programlisting></para>
        </listitem>
      </itemizedlist>
      <itemizedlist>
        <listitem>
          <para>Display information of a particular master slave session using the following command: 
</para>
          <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> status</command> 
</para>
          <para>For example, to display information of Volume1 and example.com:/data/remote_dir
</para>
          <para><command># gluster volume geo-replication Volume1 example.com:/data/remote_dir status</command> 
</para>
          <para>The status of the geo-replication between Volume1 and example.com:/data/remote_dir is displayed.</para>
        </listitem>
        <listitem>
          <para>Display information of all geo-replication sessions belonging to a master</para>
          <para><command># gluster volume geo-replication MASTER status</command> 
</para>
          <para>For example, to display information of Volume1</para>
          <para><programlisting># gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 ssh://example.com:gluster://127.0.0.1:remove_volume  OK

Volume1 ssh://example.com:file:///data/remote_dir  OK</programlisting></para>
          <para>The status of a session could be one of the following four:</para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Starting</emphasis>: This is the initial phase of the Geo-replication session; it remains in this state for a minute, to make sure no abnormalities are present.</para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">OK</emphasis>: The geo-replication session is in a stable state.</para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Faulty</emphasis>: The geo-replication session has witnessed some abnormality and the situation has to be investigated further. For further information, see <xref linkend="chap-Administration_Guide-Troubleshooting"/> section.</para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Corrupt</emphasis>: The monitor thread which is monitoring the geo-replication session has died. This situation should not occur normally, if it persists contact Red Hat Support<ulink url="www.redhat.com/support/"/>.</para>
        </listitem>
      </itemizedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Starting-Configure">
      <title>Configuring Geo-replication</title>
      <para>To configure Gluster Geo-replication </para>
      <itemizedlist>
        <listitem>
          <para>Use the following command at the Gluster command line:
</para>
          <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> config [options]</command>
</para>
          <para>For more information about the options, see <xref linkend="chap-Administration_Guide-Com_Ref"/>.
</para>
          <para>For example:
</para>
          <para>To view list of all option/value pair, use the following command:
</para>
          <para><command># gluster volume geo-replication Volume1 example.com:/data/remote_dir config</command>
</para>
        </listitem>
      </itemizedlist>
    </section>
    <section id="chap-Administration_Guide-Geo_Rep-Starting-Stop">
      <title>Stopping Geo-replication</title>
      <para>You can use the gluster command to stop Gluster Geo-replication (syncing of data from Master to Slave) in your environment. </para>
      <para><emphasis role="bold">To stop Gluster Geo-replication</emphasis> </para>
      <itemizedlist>
        <listitem>
          <para>Stop geo-replication between the hosts using the following command:
</para>
          <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> stop </command></para>
          <para>For example:
</para>
          <para><programlisting># gluster volume geo-replication Volume1 example.com:/data/remote_dir stop
Stopping geo-replication session between Volume1 and
example.com:/data/remote_dir has been successful</programlisting></para>
          <para>See <xref linkend="chap-Administration_Guide-Com_Ref"/> for more information about the gluster command.
</para>
        </listitem>
      </itemizedlist>
    </section>
  </section>
  <section id="chap-Administration_Guide-Geo_Rep-Restoring_Data">
    <title>Restoring Data from the Slave</title>
    <para>You can restore data from the slave to the master volume, whenever the master volume becomes faulty for reasons like hardware failure.
</para>
    <para>The example in this section assumes that you are using the Master Volume (Volume1) with the following configuration: 
</para>
    <para><programlisting>machine1# gluster volume info
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: machine1:/export/dir16
Brick2: machine2:/export/dir16
Options Reconfigured:
geo-replication.indexing: on</programlisting></para>
    <para>The data is syncing from master volume (Volume1) to slave directory (example.com:/data/remote_dir). To view the status of this geo-replication session run the following command on Master: </para>
    <programlisting># gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status

MASTER    SLAVE                             STATUS
______    ______________________________    ____________
Volume1  root@example.com:/data/remote_dir   OK</programlisting>
    <para><emphasis role="bold">Before Failure</emphasis> 
</para>
    <para>Assume that the Master volume had 100 files and was mounted at /mnt/gluster on one of the client machines (client). Run the following command on Client machine to view the list of files:
</para>
    <para><programlisting>client# ls /mnt/gluster | wc –l
100</programlisting></para>
    <para>The slave directory (example.com) will have same data as in the master volume and same can be viewed by running the following command on slave:
</para>
    <para><programlisting>example.com# ls /data/remote_dir/ | wc –l
100</programlisting></para>
    <para><emphasis role="bold">After Failure</emphasis>
</para>
    <para>If one of the bricks (machine2) fails, then the status of Geo-replication session is changed from &quot;OK&quot; to &quot;Faulty&quot;. To view the status of this geo-replication session run the following command on Master: 
</para>
    <programlisting># gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status

MASTER    SLAVE                              STATUS
______    ______________________________     ____________
Volume1   root@example.com:/data/remote_dir  Faulty</programlisting>
    <para>Machine2 is failed and now you can see discrepancy in number of files between master and slave. Few files will be missing from the master volume but they will be available only on slave as shown below.
</para>
    <para>Run the following command on Client:
 </para>
    <para><programlisting>client # ls /mnt/gluster | wc –l
52</programlisting></para>
    <para>Run the following command on slave (example.com):
</para>
    <para><programlisting>Example.com# # ls /data/remote_dir/ | wc –l
100</programlisting></para>
    <para><emphasis role="bold">To restore data from the slave machine</emphasis></para>
    <orderedlist>
      <listitem>
        <para>Stop all Master&apos;s geo-replication sessions using the following command:
</para>
        <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> stop</command>
</para>
        <para>For example:
</para>
        <para><programlisting>machine1# gluster volume geo-replication Volume1
example.com:/data/remote_dir stop

Stopping geo-replication session between Volume1 &amp;
example.com:/data/remote_dir has been successful</programlisting></para>
        <para><note>
            <para>Repeat  <command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> stop </command>command on all active geo-replication sessions of master volume.</para>
          </note></para>
      </listitem>
      <listitem>
        <para>Replace the faulty brick in the master by using the following command:
</para>
        <para><command># gluster volume replace-brick <replaceable>VOLNAME BRICK NEW-BRICK</replaceable> start</command>
</para>
        <para>For example:
</para>
        <para><programlisting>machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 start
Replace-brick started successfully</programlisting></para>
      </listitem>
      <listitem>
        <para>Commit the migration of data using the following command:
</para>
        <para><command># gluster volume replace-brick <replaceable>VOLNAME BRICK NEW-BRICK</replaceable> commit force </command></para>
        <para>For example:
</para>
        <para><programlisting>machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 commit force
Replace-brick commit successful</programlisting></para>
      </listitem>
      <listitem>
        <para>Verify the migration of brick by viewing the volume info using the following command:
</para>
        <para><command># gluster volume info <replaceable>VOLNAME</replaceable></command></para>
        <para>For example:
</para>
        <para><programlisting>machine1# gluster volume info
Volume Name: Volume1
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: machine1:/export/dir16
Brick2: machine3:/export/dir16
Options Reconfigured:
geo-replication.indexing: on</programlisting></para>
      </listitem>
      <listitem>
        <para>Run rsync command manually to sync data from slave to master volume&apos;s client (mount point).
</para>
        <para>For example:
</para>
        <para><command>example.com# rsync -PavhS --xattrs --ignore-existing /data/remote_dir/ client:/mnt/gluster</command></para>
        <para>Verify that the data is synced by using the following command:
</para>
        <para>On master volume, run the following command:
</para>
        <para><programlisting>Client # ls | wc –l
100</programlisting></para>
        <para>On the Slave run the following command:
</para>
        <para><programlisting>example.com# ls /data/remote_dir/ | wc –l
100</programlisting></para>
        <para>Now Master volume and Slave directory is synced.
</para>
      </listitem>
      <listitem>
        <para>Restart geo-replication session from master to slave using the following command:
</para>
        <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> start </command></para>
        <para>For example:
</para>
        <para><programlisting>machine1# gluster volume geo-replication Volume1
example.com:/data/remote_dir start
Starting geo-replication session between Volume1 &amp;
example.com:/data/remote_dir has been successful</programlisting></para>
      </listitem>
    </orderedlist>
  </section>
  <section id="chap-Administration_Guide-Geo_Rep-Best_Practices">
    <title>Best Practices</title>
    <para><emphasis role="bold">Manually Setting Time </emphasis></para>
    <para>If you have to change the time on your bricks manually, then you must set uniform time on all bricks. This avoids the out-of-time sync issue described in <xref linkend="chap-Administration_Guide-Geo_Rep-Preparation-Settingup_Environment"/>. Setting time backward corrupts the geo-replication index, so the recommended way to set the time manually is: 
</para>
    <orderedlist>
      <listitem>
        <para>Stop geo-replication between the master and slave using the following command:
</para>
        <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE</replaceable> sto</command>p
</para>
      </listitem>
      <listitem>
        <para>Stop the geo-replication indexing using the following command:
</para>
        <para><command># gluster volume set <replaceable>MASTER</replaceable> geo-replication.indexing of</command>f</para>
      </listitem>
      <listitem>
        <para>Set uniform time on 
 all bricks.s</para>
      </listitem>
      <listitem>
        <para>Restart your geo-replication sessions by using the following command: 
</para>
        <para><command># gluster volume geo-replication <replaceable>MASTER SLAVE </replaceable>start </command></para>
      </listitem>
    </orderedlist>
    <para><emphasis role="bold">Running Geo-replication commands in one system</emphasis>
</para>
    <para>It is advisable to run the geo-replication commands in one of the bricks in the trusted storage pool. This is because, the log files for the geo-replication session would be stored in the *Server* where the Geo-replication start is initiated. Hence it would be easier to locate the log-files when required.
</para>
    <para><emphasis role="bold">Isolation </emphasis></para>
    <para>Geo-replication slave operation is not sandboxed as of now and is ran as a privileged service. So for the security reason, it is advised to create a sandbox environment (dedicated machine / dedicated virtual machine / chroot/container type solution) by the administrator to run the geo-replication slave in it. Enhancement in this regard will be available in follow-up minor release.
</para>
  </section>
</chapter>