5 Administering an Active Standby Pair with Cache Groups

This chapter describes how to administer an active standby pair that replicates cache groups.

For information about managing failover and recovery automatically, see Chapter 6, "Using Oracle Clusterware to Manage Active Standby Pairs".

This chapter includes the following topics:

Active standby pairs with cache groups
Setting up an active standby pair with a read-only cache group
Setting up an active standby pair with an AWT cache group
Recovering from a failure of the active master data store
Recovering from a failure of the standby master data store
Recovering from the failure of a subscriber data store
Reversing the roles of the active and standby master data stores
Detection of dual active master data stores
Changing the configuration of an active standby pair with cache groups
Using a disaster recovery subscriber in an active standby pair

Active standby pairs with cache groups

An active standby pair that replicates a read-only cache group or an asynchronous writethrough (AWT) cache group can change the role of the cache group automatically as part of failover and recovery. This helps ensure high availability of cache instances with minimal data loss. See "Replicating an AWT cache group" and "Replicating a read-only cache group".

You can also create a special disaster recovery read-only subscriber when you set up active standby replication of an AWT cache group. This special subscriber, located at a remote disaster recovery site, can propagate updates to a second Oracle database, also located at the disaster recovery site. See "Using a disaster recovery subscriber in an active standby pair".

You cannot use an active standby pair to replicate synchronous writethrough (SWT) cache groups. If you are using an active standby pair to replicated a data store with SWT cache groups, you must either drop or exclude the SWT cache groups.

Setting up an active standby pair with a read-only cache group

This section describes how to set up an active standby pair that replicates cache tables in a read-only cache group. The active standby pair used as an example in this section is not a cache grid member.

To set up an active standby pair that replicates a local read-only cache group, complete the following tasks:

Create a cache administration user in the Oracle database. See "Create users in the Oracle database" in Oracle In-Memory Database Cache User's Guide.
Create a data store. See "Create a DSN for a TimesTen database" in Oracle In-Memory Database Cache User's Guide.
Set the cache administration user ID and password by calling the ttCacheUidPwdSet built-in procedure. See "Set the cache administration user name and password in the TimesTen database" in Oracle In-Memory Database Cache User's Guide. For example:
```
Command> call ttCacheUidPwdSet('orauser','orapwd');
```
Start the cache agent on the data store. Use the ttCacheStart built-in procedure or the ttAdmin -cachestart utility.
```
Command> call ttCacheStart;
```

Use the CREATE CACHE GROUP statement to create the read-only cache group. For example:

Command> CREATE READONLY CACHE GROUP readcache
       > AUTOREFRESH INTERVAL 5 SECONDS
       > FROM oratt.readtab
       > (keyval NUMBER NOT NULL PRIMARY KEY, str VARCHAR2(32));

Ensure that the AUTOREFRESH STATE is set to PAUSED. The autorefresh state is PAUSED by default after cache group creation. You can verify the autorefresh state by executing the ttIsql cachegroups command:
```
Command> cachegroups;
```
Create the replication scheme using the CREATE ACTIVE STANDBY PAIR statement.

For example, suppose master1 and master2 are defined as the master data stores. sub1 and sub2 are defined as the subscriber data stores. The data stores reside on node1, node2, node3, and node4. The return service is RETURN RECEIPT. The replication scheme can be specified as follows:
```
Command> CREATE ACTIVE STANDBY PAIR master1 ON "node1", master2 ON "node2"
       > RETURN RECEIPT
       > SUBSCRIBER sub1 ON "node3", sub2 ON "node4"
       > STORE master1 ON "node1" PORT 21000 TIMEOUT 30
       > STORE master2 ON "node2" PORT 20000 TIMEOUT 30;
```
Set up the replication agent policy for master1 and start the replication agent. See "Starting and stopping the replication agents".
Set the replication state to ACTIVE by calling the ttRepStateSet built-in procedure on the active master data store (master1). For example:
```
Command> call ttRepStateSet('ACTIVE');
```
Load the cache group by using the LOAD CACHE GROUP statement. This starts the autorefresh process. For example:
```
Command> LOAD CACHE GROUP readcache COMMIT EVERY 256 ROWS;
```
As the instance administrator, duplicate the active master data store (master1) to the standby master data store (master2). Use the ttRepAdmin -duplicate utility with the -keepCG option to preserve the cache group. Alternatively, you can use the ttRepDuplicateEx C function to duplicate the data store. See "Duplicating a data store". In this example, cacheuser is a TimesTen user with ADMIN privilege.
```
ttRepAdmin -duplicate -from master1 -host node1 -uid cacheuser -pwd cachepwd -keepCG -cacheuid orauser -cacheuid orapwd "DSN=master2;UID=;PWD="
```
Set up the replication agent policy on master2 and start the replication agent. See "Starting and stopping the replication agents".
The standby master database enters the STANDBY state automatically. Wait for master2 to enter the STANDBY state. Call the ttRepStateGet built-in procedure to check the state of master2. For example:
```
Command> call ttRepStateGet;
```
Start the cache agent for master2 using the ttCacheStart built-in procedure or the ttAdmin -cacheStart utility. For example:
```
Command> call ttCacheStart;
```
As the instance administrator, duplicate the subscribers (sub1 and sub2) from the standby master data store (master2). Use the -noKeepCG command line option with ttRepAdmin -duplicate to convert the cache tables to normal TimesTen tables on the subscribers. See "Duplicating a data store". For example:
```
ttRepAdmin -duplicate -from master2 -host node2 -uid cacheuser -pwd cachepwd -nokeepCG "DSN=sub1;UID=;PWD="
```
Set up the replication agent policy on the subscribers and start the replication agent on each of the subscriber stores. See "Starting and stopping the replication agents".

Setting up an active standby pair with an AWT cache group

For detailed instructions for setting up an active standby pair with a global AWT cache group, see "Replicating cache tables" in Oracle In-Memory Database Cache User's Guide. The active standby pair in that section is a cache grid member.

Recovering from a failure of the active master data store

This section includes the following topics:

Recovering when the standby master data store is ready
Recovering when the standby master data store is not ready
Failing back to the original nodes

Recovering when the standby master data store is ready

This section describes how to recover the active master data store when the standby master data store is available and synchronized with the active master data store. It includes the following topics:

When replication is return receipt or asynchronous
When replication is return twosafe

When replication is return receipt or asynchronous

Complete the following tasks:

Stop the replication agent on the failed data store if it has not already been stopped.
On the standby master data store, execute ttRepStateSet('ACTIVE'). This changes the role of the data store from STANDBY to ACTIVE. If you are replicating an read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this action automatically causes the AUTOREFRESH state to change from PAUSED to ON for this data store.
On the new active master data store, execute ttRepStateSave('FAILED', 'failed_store','host_name'), where failed_store is the former active master data store that failed. This step is necessary for the new active master data store to replicate directly to the subscriber data stores.
Stop the cache agent on the failed data store if it is not already stopped.
Destroy the failed data store.
Duplicate the new active master data store to the new standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG -recoveringNode command line options with ttRepAdmin in order to preserve the cache group. See "Duplicating a data store".
Set up the replication agent policy on the new standby master data store and start the replication agent. See "Starting and stopping the replication agents".
Start the cache agent on the new standby master data store.

The standby master data store contacts the active master data store. The active master data store stops sending updates to the subscribers. When the standby master data store is fully synchronized with the active master data store, then the standby master data store enters the STANDBY state and starts sending updates to the subscribers.The new standby master data store takes over processing of the cache group automatically when it enters the STANDBY state.

Note:

You can verify that the standby master data has entered the STANDBY state by using the ttRepStateGet built-in procedure.

When replication is return twosafe

Complete the following tasks:

On the standby master data store, execute ttRepStateSet('ACTIVE'). This changes the role of the data store from STANDBY to ACTIVE. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this action automatically causes the AUTOREFRESH state to change from PAUSED to ON for this data store.
On the new active master data store, execute ttRepStateSave('FAILED', 'failed_store','host_name'), where failed_store is the former active master data store that failed. This step is necessary for the new active master data store to replicate directly to the subscriber data stores.
Connect to the failed data store. This triggers recovery from the local transaction logs. If data store recovery fails, you must continue from Step 5 of the procedure for recovering when replication is return receipt or asynchronous. See "When replication is return receipt or asynchronous". If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, the autorefresh state is automatically set to PAUSED.
Verify that the replication agent for the failed data store has restarted. If it has not restarted, then start the replication agent. See "Starting and stopping the replication agents".
Verify that the cache agent for the failed data store has restarted. If it has not restarted, then start the cache agent.

When the active master data store determines that it is fully synchronized with the standby master data store, then the standby master store enters the STANDBY state and starts sending updates to the subscribers. The new standby master data store takes over processing of the cache group automatically when it enters the STANDBY state.

Note:

You can verify that the standby master data has entered the STANDBY state by using the ttRepStateSet built-in procedure.

Recovering when the standby master data store is not ready

Consider the following scenarios:

The standby master data store fails. The active master data store fails before the standby comes back up or before the standby has been synchronized with the active master data store.
The active master data store fails. The standby master data store becomes ACTIVE, and the rest of the recovery process begins. (See "Recovering from a failure of the active master data store".) The new active master data store fails before the new standby master data store is fully synchronized with it.

In both scenarios, the subscribers may have had more changes applied than the standby master data store.

When the active master data store fails and the standby master data store has not applied all of the changes that were last sent from the active master data store, there are two choices for recovery:

Recover the active master data store from the local transaction logs.
Recover the standby master data store from the local transaction logs.

The choice depends on which data store is available and which is more up to date.

Recover the active master data store

Connect to the failed active data store. This triggers recovery from the local transaction logs. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, the autorefresh state is automatically set to PAUSED.
Verify that the replication agent for the failed active data store has restarted. If it has not restarted, then start the replication agent. See "Starting and stopping the replication agents".
Execute ttRepStateSet('ACTIVE') on the newly recovered store. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this action automatically causes the AUTOREFRESH state to change from PAUSED to ON for this data store.
Verify that the cache agent for the failed data store has restarted. If it has not restarted, then start the cache agent.
Duplicate the active master data store to the standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG command line option with ttRepAdmin in order to preserve the cache group. "Duplicating a data store".
Set up the replication agent policy on the standby master data store and start the replication agent. See "Starting and stopping the replication agents".
Wait for the standby master data store to enter the STANDBY state. Use the ttRepStateGet procedure to check the state.
Start the cache agent for on the standby master data store using the ttCacheStart procedure or the ttAdmin -cacheStart utility.
Duplicate all of the subscribers from the standby master data store. See "Copying a master data store to a subscriber". Use the -noKeepCG command line option with ttRepAdmin in order to convert the cache group to regular TimesTen tables on the subscribers.
Set up the replication agent policy on the subscribers and start the agent on each of the subscriber stores. See "Starting and stopping the replication agents".

Recover the standby master data store

Connect to the failed standby data store. This triggers recovery from the local transaction logs. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, the autorefresh state is automatically set to PAUSED.
If the replication agent for the standby data store has automatically restarted, you must stop the replication agent. See "Starting and stopping the replication agents".
If the cache agent has automatically restarted, stop the cache agent.
Drop the replication configuration using the DROP ACTIVE STANDBY PAIR statement.
Drop and re-create all cache groups using the DROP CACHE GROUP and CREATE CACHE GROUP statements.
Re-create the replication configuration using the CREATE ACTIVE STANDBY PAIR statement.
Set up the replication agent policy and start the replication agent. See "Starting and stopping the replication agents".
Execute ttRepStateSet('ACTIVE') on the master data store, giving it the ACTIVE role. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this action automatically causes the AUTOREFRESH state to change from PAUSED to ON for this data store.
Start the cache agent on the active master data store.
Duplicate the active master data store to the standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG command line option with ttRepAdmin in order to preserve the cache group. "Duplicating a data store".
Set up the replication agent policy on the standby master data store and start the replication agent. See "Starting and stopping the replication agents".
Wait for the standby master data store to enter the STANDBY state. Use the ttRepStateGet procedure to check the state.
Start the cache agent for the standby master data store using the ttCacheStart procedure or the ttAdmin -cacheStart utility.
Duplicate all of the subscribers from the standby master data store. See "Copying a master data store to a subscriber". Use the -noKeepCG command line option with ttRepAdmin in order to convert the cache group to regular TimesTen tables on the subscribers.
Set up the replication agent policy on the subscribers and start the agent on each of the subscriber stores. See "Starting and stopping the replication agents".

Failing back to the original nodes

After a successful failover, you may wish to fail back so that the active master data store and the standby master data store are on their original nodes. See "Reversing the roles of the active and standby master data stores" for instructions.

Recovering from a failure of the standby master data store

To recover from a failure of the standby master data store, complete the following tasks:

Detect the standby master data store failure.
If return twosafe service is enabled, the failure of the standby master data store may prevent a transaction in progress from being committed on the active master data store, resulting in error 8170, "Receipt or commit acknowledgement not returned in the specified timeout interval". If so, then call the ttRepStateSet procedure with a localAction parameter of 2 (COMMIT) and commit the transaction again. For example:
```
call ttRepStateSet( null, null, 2);
commit;
```
Execute ttRepStateSave('FAILED','standby_store','host_name') on the active master data store. After this, as long as the standby data store is unavailable, updates to the active data store are replicated directly to the subscriber data stores. Subscriber stores may also be duplicated directly from the active master.
If the replication agent for the standby data store has automatically restarted, stop the replication agent. See "Starting and stopping the replication agents".
If the cache agent has automatically restarted, stop the cache agent.
Recover the standby master data store in one of the following ways:
- Connect to the standby master data store. This triggers recovery from the local transaction logs.
- Duplicate the standby master data store from the active master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG -recoveringNode command line options with ttRepAdmin in order to preserve the cache group.See "Duplicating a data store".
The amount of time that the standby master data store has been down and the amount of transaction logs that need to be applied from the active master data store determine the method of recovery that you should use.
Set up the replication agent policy and start the replication agent. See "Starting and stopping the replication agents".
Start the cache agent.

The standby master data store enters the STANDBY state after the active master data store determines that the two master data stores have been synchronized.

Note:

You can verify that the standby master data has entered the STANDBY state by using the ttRepStateGet procedure

Recovering from the failure of a subscriber data store

If a subscriber data store fails, then you can recover it by one of the following methods:

Connect to the failed subscriber. This triggers recovery from the local transaction logs. Start the replication agent and let the subscriber catch up.
Duplicate the subscriber from the standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -noKeepCG command line option with ttRepAdmin in order to convert the cache group to normal TimesTen tables on the subscriber.

If the standby master data store is down or in recovery, then duplicate the subscriber from the active master data store.

After the subscriber data store has been recovered, then set up the replication agent policy and start the replication agent. See "Starting and stopping the replication agents".

Reversing the roles of the active and standby master data stores

To change the active master data store's role to that of a standby master data store and vice versa:

Pause any applications that are generating updates on the current active master data store.
Execute ttRepSubscriberWait on the active master data store, with the DSN and host of the current standby data store as input parameters. This ensures that all updates have been transmitted to the current standby master data store.
Stop the replication agent on the current active master data store. See "Starting and stopping the replication agents".
Stop the cache agent on the active master data store.
Execute ttRepDeactivate on the current active master data store. This puts the store in the IDLE state. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this action automatically causes the AUTOREFRESH state to change from ON to PAUSED for this data store.
Execute ttRepStateSet('ACTIVE') on the current standby master data store. This store now acts as the active master data store in the active standby pair. If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, this automatically causes the AUTOREFRESH state to change from PAUSED to ON for this data store.
Configure the replication agent policy as needed and start the replication agent on the old active master data store. Use the ttRepStateGet procedure to determine when the data store's state has changed from IDLE to STANDBY. The data store now acts as the standby master data store in the active standby pair.
Start the cache agent on the old active master data store.
Resume any applications that were paused in Step 1.

Detection of dual active master data stores

See "Detection of dual active master data stores". There is no difference for active standby pairs that replicate cache groups.

Changing the configuration of an active standby pair with cache groups

You can change an active standby pair by:

Adding or dropping a subscriber data store
Altering store attributes - Only the PORT and TIMEOUT attributes can be set for subscribers. The RELEASE clause cannot be set for any data store in an active standby pair.
Including tables or cache groups in the active standby pair
Excluding tables or cache groups from the active standby pair

Make these changes on the active master data store. After you have changed the replication scheme on the active master data store, it no longer replicates updates to the standby master data store or to the subscribers. You must re-create the standby master data store and the subscribers and restart the replication agents.

Use the ALTER ACTIVE STANDBY PAIR statement to change the active standby pair.

To change an active standby pair, complete the following tasks:

Stop the replication agent on the active master data store. See "Starting and stopping the replication agents".
Stop the cache agent on the active master data store.
Use the ALTER ACTIVE STANDBY PAIR statement to make changes to the replication scheme.
Start the replication agent on the active master data store. See "Starting and stopping the replication agents".
Start the cache agent on the active master data store.
Destroy the standby master data store and the subscribers.
Duplicate the active master data store to the standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG command line option with ttRepAdmin in order to preserve the cache group. See "Duplicating a data store".
Set up the replication agent policy on the standby master data store and start the replication agent. See "Starting and stopping the replication agents".
Wait for the standby master data store to enter the STANDBY state. Use the ttRepStateGet procedure to check the state.
Start the cache agent for the standby master data store using the ttCacheStart procedure or the ttAdmin -cacheStart utility.
Duplicate all of the subscribers from the standby master data store. See "Copying a master data store to a subscriber". Use the -noKeepCG command line option with ttRepAdmin in order to convert the cache group to regular TimesTen tables on the subscribers. See "Duplicating a data store".
Set up the replication agent policy on the subscribers and start the agent on each of the subscriber stores. See "Starting and stopping the replication agents".

Example 5-1 Adding a subscriber to an active standby pair

Add a subscriber data store to the active standby pair.

ALTER ACTIVE STANDBY PAIR
ADD SUBSCRIBER sub1;

Example 5-2 Dropping subscribers from an active standby pair

Drop subscriber data stores from the active standby pair.

ALTER ACTIVE STANDBY PAIR
DROP SUBSCRIBER sub1
DROP SUBSCRIBER sub2;

Example 5-3 Changing the PORT and TIMEOUT settings for subscribers

Alter the PORT and TIMEOUT settings for subscribers rep3 and rep4.

ALTER ACTIVE STANDBY PAIR
ALTER STORE sub1 SET PORT 23000 TIMEOUT 180
ALTER STORE sub2 SET PORT 23000 TIMEOUT 180;

Example 5-4 Adding tables and a cache group to an active standby pair

Add two tables and a cache group to the active standby pair.

ALTER ACTIVE STANDBY PAIR
INCLUDE TABLE tab1, tab2
INCLUDE CACHE GROUP cg0;

Using a disaster recovery subscriber in an active standby pair

TimesTen active standby pair replication provides high availability by allowing for fast switching between data stores within a data center. This includes the ability to automatically change which data store propagates changes to an Oracle database using AWT cache groups. However, for additional high availability across data centers, you may require the ability to recover from a failure of an entire site, which can include a failure of both TimesTen master data stores in the active standby pair as well as the Oracle database used for the cache groups.

You can recover from a complete site failure by creating a special disaster recovery read-only subscriber as part of the active standby pair replication scheme. The standby master data store sends updates to cache group tables on the read-only subscriber. This special subscriber is located at a remote disaster recovery site and can propagate updates to a second Oracle database, also located at the disaster recovery site. The disaster recovery subscriber can take over as the active master in a new active standby pair at the disaster recovery site if the primary site suffers a complete failure. Any applications may then connect to the disaster recovery site and continue operating, with minimal interruption of service.

Requirements for using a disaster recovery subscriber with an active standby pair

To use a disaster recovery subscriber, you must:

Be using an active standby pair configuration with AWT cache groups at the primary site.
Have a continuous WAN connection from the primary site to the disaster recovery site. This connection should have at least enough bandwidth to guarantee that the normal volume of transactions can be replicated to the disaster recovery subscriber at a reasonable pace.
Have an Oracle database configured at the disaster recovery site to include tables with the same schema as the database at the primary site. Note that this database is intended only for capturing the replicated updates from the primary site, and if any data exists in tables written to by the cache groups when the disaster recovery subscriber is created, that data is deleted.
Have the same cache group administrator user ID and password at both the primary and the disaster recovery site.

Though it is not absolutely required, you should have a second TimesTen data store configured at the disaster recovery site. This data store can take on the role of a standby master data store, in the event that the disaster recovery subscriber is promoted to an active master data store after the primary site fails.

Rolling out a disaster recovery subscriber

To create a disaster recovery subscriber, follow these steps:

Create an active standby pair with AWT cache groups at the primary site.
Create the disaster recovery subscriber at the disaster recovery site using the ttRepAdmin utility with the -duplicate and -cacheInitDR options. You must also specify the cache group administrator and password for the Oracle database at the disaster recovery site using the -cacheUid and -cachePwd options.

If your data store includes multiple cache groups, you may improve the efficiency of the duplicate operation by using the -nThreads option to specify the number of threads that are spawned to flush the cache groups in parallel. Each thread flushes an entire cache group to Oracle and then moves on to the next cache group, if any remain to be flushed. If a value is not specified for -nThreads, only one flushing thread is spawned.

For example, duplicate the standby master data store mast2, on the system with the host name primary and the cache user ID system and password manager, to the disaster recovery subscriber drsub, and using two cache group flushing threads. This example assumes that you have a user ttuser with password ttuser with the ADMIN privilege.
```
ttRepAdmin -duplicate -from mast2 -host primary -uid ttuser -pwd ttuser -cacheInitDR -nThreads 2 -cacheUid system -cachePwd manager drsub
```
If you use the ttRepDuplicateEx function in C, you must set the TT_REPDUPE_INITCACHEDR flag in ttRepDuplicateExArg.flags and may optionally specify a value for ttRepDuplicateExArg.nThreads4InitDR:
```
int                 rc;
ttUtilHandle        utilHandle;
ttRepDuplicateExArg arg;
memset( &arg, 0, sizeof( arg ) );
arg.size = sizeof( ttRepDuplicateExArg );
arg.flags = TT_REPDUPE_INITCACHEDR;
arg.nThreads4InitDR = 2;
arg.uid="ttuser"
arg.pwd="ttuser"
arg.cacheuid = "system";
arg.cachepwd = "manager";
arg.localHost = "disaster";
rc = ttRepDuplicateEx( utilHandle, "DSN=drsub",
                       "mast2", "primary", &arg );
```
After the subscriber is duplicated, TimesTen automatically configures the asynchronous writethrough replication scheme that propagates updates from the cache groups to the Oracle database, truncates the tables in the Oracle database that correspond to the cache groups in TimesTen, and then flushes all of the data in the cache groups to the Oracle database.
If you wish to set the failure threshold for the disaster recovery subscriber, call the ttCacheAWTThresholdSet built-in procedure and specify the number of transaction log files that can accumulate before the disaster recovery subscriber is considered either dead or too far behind to catch up.

If one or both master data stores had a failure threshold configured before the disaster recovery subscriber was created, then the disaster recovery subscriber inherits the failure threshold value when it is created with the ttRepAdmin -duplicate -initCacheDR command. If the master data stores have different failure thresholds, then the higher value is used for the disaster recovery subscriber.

For more information about the failure threshold, see "Setting the log failure threshold".
Start the replication agent for the disaster recovery subscriber using the ttRepStart procedure or the ttAdmin command with the option -repstart. For example:
```
ttAdmin -repstart drsub
```
Updates are now replicated from the standby master data store to the disaster recovery subscriber, which then propagates the updates to the Oracle database at the disaster recovery site.

Switching over to the disaster recovery site

When the primary site has failed, you can switch over to the disaster recovery site in one of two ways. If your goal is to minimize risk of data loss at the disaster recovery site, you may roll out a new active standby pair using the disaster recovery subscriber as the active master data store. If the goal is to absolutely minimize the downtime of your applications, at the risk of data loss if the disaster recovery data store later fails, you may instead choose to drop the replication scheme from the disaster recovery subscriber and use it as a single non-replicating data store. You may deploy an active standby pair at the disaster recovery site later.

Creating a new active standby pair after switching to the disaster recovery site

Any read-only applications may be redirected to the disaster recovery subscriber immediately. Redirecting applications that make updates to the data store must wait until Step 7.
Ensure that all of the recent updates to the cache groups have been propagated to the Oracle database using the ttRepSubscriberWait procedure or the ttRepAdmin command with the -wait option.
```
ttRepSubscriberWait( null, null, '_ORACLE', null, 600 );
```
If ttRepSubscriberWait returns 0x01, indicating a timeout, you may need to investigate to determine why the cache groups are not finished propagating before continuing to Step 3.
Stop the replication agent on the disaster recovery subscriber using the ttRepStop procedure or the ttAdmin command with the -repstop option. For example, to stop the replication agent for the subscriber drsub, use:
```
call ttRepStop;
```
Drop the active standby pair replication scheme on the subscriber using the DROP ACTIVE STANDBY PAIR statement. For example:
```
DROP ACTIVE STANDBY PAIR;
```
Create a new active standby pair replication scheme using the CREATE ACTIVE STANDBY PAIR statement, specifying the disaster recovery subscriber as the active master data store. For example, to create a new active standby pair with the former subscriber drsub as the active master and the new data store drstandby as the standby master, and using the return twosafe return service, use:
```
CREATE ACTIVE STANDBY PAIR drsub, drstandby RETURN TWOSAFE;
```
Set the new active standby master data store to the ACTIVE state using the ttRepStateSet procedure. For example, on the data store drsub in this example, execute:
```
call ttRepStateSet( 'ACTIVE' );
```
Any applications which must write to the TimesTen data store may now be redirected to the new active master data store.
If you are replicating a read-only cache group or an AWT cache group with the AUTOREFRESH attribute, load the cache group using the LOAD CACHE GROUP statement to begin the autorefresh process. You may also load the cache group if you are replicating an AWT cache group without the AUTOREFRESH attribute, although it is not required.
Duplicate the active master data store to the standby master data store. You can use either the ttRepAdmin -duplicate utility or the ttRepDuplicateEx C function to duplicate a data store. Use the -keepCG command line option with ttRepAdmin in order to preserve the cache group. See "Duplicating a data store".
Set up the replication agent policy on the standby master data store and start the replication agent. See "Starting and stopping the replication agents".
Wait for the standby master data store to enter the STANDBY state. Use the ttRepStateGet procedure to check the state.
Start the cache agent for the standby master data store using the ttCacheStart procedure or the ttAdmin -cacheStart utility.
Duplicate all of the subscribers from the standby master data store. See "Copying a master data store to a subscriber". Use the -noKeepCG command line option with ttRepAdmin in order to convert the cache group to regular TimesTen tables on the subscribers.
Set up the replication agent policy on the subscribers and start the agent on each of the subscriber stores. See "Starting and stopping the replication agents".

Switching over to a single data store

Any read-only applications may be redirected to the disaster recovery subscriber immediately. Redirecting applications that make updates to the data store must wait until Step 5.
Stop the replication agent on the disaster recovery subscriber using the ttRepStop procedure or the ttAdmin command with the -repstop option. For example, to stop the replication agent for the subscriber drsub, use:
```
call ttRepStop;
```
Drop the active standby pair replication scheme on the subscriber using the DROP ACTIVE STANDBY PAIR statement. For example:
```
DROP ACTIVE STANDBY PAIR;
```
Although there is no longer an active standby pair configured, AWT cache groups require the replication agent to be started. Start the replication agent on the data store using the ttRepStart procedure or the ttAdmin command with the -repstart option. For example, to start the replication agent for the data store drsub, use:
```
call ttRepStart;
```
Any applications which must write to a TimesTen data store may now be redirected to the this data store.

Note:
You may choose to roll out an active standby pair at the disaster recovery site at a later time. You may do this by following the steps in "Creating a new active standby pair after switching to the disaster recovery site", starting at Step 2 and skipping Step 4.

Returning to the original configuration at the primary site

When the primary site is usable again, you may wish to move the working active standby pair from the disaster recovery site back to the primary site. You can do this with a minimal interruption of service by reversing the process that was used to create and switch over to the original disaster recovery site. Follow these steps:

Destroy original active master data store at the primary site, if necessary, using the ttDestroy utility. For example, to destroy a data store called mast1, use:
```
ttDestroy mast1
```
Create a disaster recovery subscriber at the primary site, following the steps detailed in "Rolling out a disaster recovery subscriber". Use the original active master data store for the new disaster recovery subscriber.
Switch over to the new disaster recovery subscriber at primary site, as detailed in "Switching over to the disaster recovery site". Roll out the standby master data store as well.
Roll out a new disaster recovery subscriber at the disaster recovery site, as detailed in "Rolling out a disaster recovery subscriber".