11 Conflict Resolution and Failure Recovery

This chapter describes:

Replication conflict detection and resolution
Managing data store failover and recovery

Replication conflict detection and resolution

Tables in data stores configured in a bidirectional replication scheme, as described in "Hot standby configuration", may be subject to replication conflicts. A replication conflict occurs when applications on bidirectionally replicated data stores initiate an UPDATE, INSERT or DELETE operation on the same data item at the same time. If no special steps are taken, each data store can end up in disagreement with the last update made by the other data store.

Three different types of replication conflicts can occur:

Update conflicts: This type of conflict occurs when concurrently running transactions at different stores make simultaneous UPDATE requests on the same row in the same table, and install different values for one or more columns.
Uniqueness conflicts: This type of conflict occurs when concurrently running transactions at different stores make simultaneous INSERT requests for a row in the same table that has the same primary or unique key, but different values for one or more other columns.
Delete conflicts: This type of conflict occurs when a transaction at one store deletes a row while a concurrent transaction at another store simultaneously updates or inserts the same row. Currently, TimesTen can detect delete/update conflicts, but cannot detect delete/insert conflicts. TimesTen cannot resolve either type of delete conflict.

See "Conflict reporting" for example reports generated by TimesTen upon detecting update, uniqueness, and delete conflicts.

Note:

TimesTen does not detect conflicts involving TRUNCATE TABLE statements.

Update and insert conflicts

Figure 11-1 shows the results from an update conflict, which would occur for the value X under the following circumstances:

Steps	On data store A	On data store B
Initial condition	X is 1	X is 1
The application on each data store updates X simultaneously	Set X=2	Set X=100
The replication agent on each data store sends its update to the other	Replicate X to data store B	Replicate X to data store A
Each data store now has the other's update	Replication says to set X=100	Replication says to set X=2

Note:

Uniqueness conflicts resulting from conflicting inserts follow a similar pattern as update conflicts, only the conflict involves the whole row.

Figure 11-1 Update conflict

Description of "Figure 11-1 Update conflict"

If update or insert conflicts remain unchecked, the master and subscriber data stores fall out of synchronization with each other. It may be difficult or even impossible to determine which data store is correct.

With update conflicts, it is possible for a transaction to update many data items but have a conflict on a few of them. Most of the transaction's effects survive the conflict, with only a few being overwritten by replication. If you decide to ignore such conflicts, the transactional consistency of the application data is compromised.

If an update conflict occurs, and if the updated columns for each version of the row are different, then the non-primary key fields for the row may diverge between the replicated tables.

Note:

Within a single data store, update conflicts are prevented by the locking protocol: only one transaction at a time can update a specific row in the data store. However, update conflicts can occur in replicated systems due to the ability of each data store to operate independently.

TimesTen replication includes timestamp-based conflict resolution to cope with simultaneous updates or inserts. Through the use of timestamp-based conflict resolution, you may be able to keep the replicated data stores synchronized and transactionally consistent.

Delete/update conflicts

Figure 11-2 shows the results from a delete/update conflict, which would occur for Row 4 under the following circumstances:

Steps	On data store A	On data store B
Initial condition	Row 4 exists	Row 4 exists
The applications issue a conflicting update and delete on Row 4 simultaneously	Update Row 4	Delete Row 4
The replication agent on each data store sends the delete or update to the other	Replicate update to data store B	Replicate delete to data store A
Each data store now has the delete or update from the other data store	Replication says to delete Row 4	Replication says to update Row 4

Figure 11-2 Update/delete conflict

Description of "Figure 11-2 Update/delete conflict"

Though TimesTen can detect and report delete/update conflicts, it cannot resolve them. Under these circumstances, the master and subscriber data stores fall out of synchronization with each other.

Though TimesTen cannot ensure synchronization between data stores following such a conflict, it does ensure that the most recent transaction is applied to each data store. If the timestamp for the delete is more recent than that for the update, the row is deleted on each data store. If the timestamp for the update is more recent than that for the delete, the row is updated on the local data store. However, because the row was deleted on the other data store, the replicated update is discarded. See "Reporting DELETE/UPDATE conflicts" for example reports.

Note:

There is an exception to this behavior when timestamp comparison is enabled on a table using UPDATE BY USER. See "User timestamp column maintenance" for details.

Timestamp resolution

For replicated tables that are subject to conflicts, create the table with a special column of type BINARY(8) to hold a timestamp value that indicates the time the row was inserted or last updated. You can then configure TimesTen to automatically insert a timestamp value into this column each time a particular row is changed, as described in "Configuring timestamp comparison".

Note:

TimesTen does not support conflict resolution between cached tables in a cache group and Oracle.

How replication computes the timestamp column depends on your system:

On UNIX systems, the timestamp value is derived from the timeval structure returned by the gettimeofday system call. This structure reports the time of day in a pair of 4-byte words to a resolution of 1 microsecond. The actual resolution of the value is system-dependent.
On Windows systems, the timestamp value is derived from the GetSystemTimeAsFileTime Win32 call. The Windows file time is reported in units of 0.1 microseconds, but effective granularity can be as coarse as 10 milliseconds.

TimesTen uses the time value returned by the system at the time the transaction performs each update as the record's INSERT or UPDATE time. Therefore, rows that are inserted or updated by a single transaction may receive different timestamp values.

When applying an update received from a master, the replication agent at the subscriber data store performs timestamp resolution in the following manner:

If the timestamp of the update record is newer than the timestamp of the stored record, TimesTen updates the row. The same rule applies to inserts. If a replicated insert is newer than an existing row, the existing row is overwritten.
If the timestamp of the update and of the stored record are equal, the update is allowed. The same rule applies to inserts.
If the timestamp of the update is older than the timestamp of the stored record, the update is discarded. The same rule applies to inserts.
If a row is deleted, no timestamp is available for comparison. Any update operations on the deleted row are discarded. However, if a row is deleted on one system, then replicated to another system that has more recently updated the row, then the replicated delete is rejected. A replicated insert operation on a deleted row is applied as an insert.
An update that cannot find the updated row is considered a delete conflict, which is reported but cannot be resolved.

Note:
If the ON EXCEPTION NO ACTION option is specified for a table, then the update, insert, or delete that fails a timestamp comparison is rejected. This may result in transactional inconsistencies should replication apply some, but not all, the actions of a transaction. If the ON EXCEPTION ROLLBACK WORK option is specified for a table, an update that fails timestamp comparison causes the entire transaction to be skipped.

Configuring timestamp comparison

To configure timestamp comparison:

Define a column in your replicated tables to hold the timestamp value.
Include a CHECK CONFLICTS clause for each TABLE element in your CREATE REPLICATION statement to identify the timestamp column, how timestamps are to be generated, what to do in the event of a conflict, and how to report conflicts.

Establishing a timestamp column in replicated tables

To use timestamp comparison on replicated tables, you must specify a nullable column of type BINARY(8) to hold the timestamp value. The timestamp column must be created along with the table as part of a CREATE TABLE statement - it cannot be added later as part of an ALTER TABLE statement. In addition, the timestamp column cannot be part of a primary key or index. Example 11-1 shows the rep.tab table contains a column named tstamp of type BINARY(8) to hold the timestamp value.

Example 11-1 Including a timestamp column when creating a table

CREATE TABLE rep.tab (col1 NUMBER NOT NULL,
                      col2 NUMBER NOT NULL,
                      tstamp BINARY(8),
                      PRIMARY KEY (col1));

If no timestamp column is defined in the replicated table, timestamp comparison cannot be performed to detect conflicts. Instead, at each site, the value of a row in the database reflects the most recent update applied to the row, either by local applications or by replication.

Configuring the CHECK CONFLICTS clause

When configuring your replication scheme, you can set up timestamp comparison for a TABLE element by including a CHECK CONFLICTS clause in the table's ELEMENT description in the CREATE REPLICATION statement.

Note:

A CHECK CONFLICT clause cannot be specified for DATASTORE elements.

The syntax of the CREATE REPLICATION statement is described in Oracle TimesTen In-Memory Database SQL Reference. Below are some examples of how CHECK CONFLICTS might be used when configuring your replication scheme.

Example 11-2 Automatic timestamp comparison

In this example, we establish automatic timestamp comparison for the bidirectional replication scheme shown in Example 7-30. The DSNs, west_dsn and east_dsn, define the westds and eastds data stores that replicate the table, repl.accounts, containing the timestamp column, tstamp. In the event of a comparison failure, discard the transaction that includes an update with the older timestamp.

CREATE REPLICATION r1
ELEMENT elem_accounts_1 TABLE accounts
  CHECK CONFLICTS BY ROW TIMESTAMP
    COLUMN tstamp
    UPDATE BY SYSTEM
    ON EXCEPTION ROLLBACK WORK
  MASTER westds ON "westcoast"
  SUBSCRIBER eastds ON "eastcoast"
ELEMENT elem_accounts_2 TABLE accounts
  CHECK CONFLICTS BY ROW TIMESTAMP
    COLUMN tstamp
    UPDATE BY SYSTEM
    ON EXCEPTION ROLLBACK WORK
  MASTER eastds ON "EASTCOAST"
  SUBSCRIBER westds ON "WESTCOAST";

When bidirectionally replicating data stores with conflict resolution, the replicated tables on each data store must be set with the same CHECK CONFLICTS attributes. If you need to disable or change the CHECK CONFLICTS settings for the replicated tables, use the ALTER REPLICATION statement described in "Eliminating conflict detection" and apply to each replicated data store.

System timestamp column maintenance

When timestamp comparison is enabled using:

CHECK CONFLICTS BY ROW TIMESTAMP
  COLUMN ColumnName
  UPDATE BY SYSTEM

TimesTen automatically maintains the value of the timestamp column using the current time returned by the underlying operating system. This is the default setting.

When you specify UPDATE BY SYSTEM, TimesTen:

Initializes the timestamp column to the current time when a new record is inserted into the table.
Updates the timestamp column to the current time when an existing record is modified.

During initial load, the timestamp column values should be left NULL, and applications should not give a value for the timestamp column when inserting or updating a row.

When you use the ttBulkCp or ttMigrate utility to save TimesTen tables, the saved rows maintain their current timestamp values. When the table is subsequently copied or migrated back into TimesTen, the timestamp column retains the values it had when the copy or migration file was created.

Note:

If you configure TimesTen for timestamp comparison after using the ttBulkCp or ttMigrate to copy or migrate your tables, the initial values of the timestamp columns remain NULL, which is considered by replication to be the earliest possible time.

User timestamp column maintenance

When timestamp comparison is enabled on a table using:

CHECK CONFLICTS BY ROW TIMESTAMP
  COLUMN ColumnName
  UPDATE BY USER

your application is responsible for maintaining timestamp values. The timestamp values used by your application can be arbitrary, but the time values cannot decrease. In cases where the user explicitly sets or updates the timestamp column, the application-provided value is used instead of the current time.

Note:

Replicated delete operations always carry a system-generated timestamp. If replication has been configured with UPDATE BY USER and an update/delete conflict occurs, the conflict is resolved by comparing the two timestamp values and the operation with the larger timestamp wins. If the basis for the user timestamp varies from that of the system-generated timestamp, the results may not be as expected. Therefore, if you expect delete conflicts to occur, use system-generated timestamps.

Local updates

To maintain synchronization of tables between replicated sites, TimesTen also performs timestamp comparisons for updates performed by local transactions. If an updated table is declared to have automatic timestamp maintenance, then updates to records that have timestamps exceeding the current system time are prohibited.

Normally, clocks on replicated systems are synchronized sufficiently to ensure that a locally updated record is given a later timestamp than that in the same record stored on the other systems. Perfect synchronization may not be possible or affordable, however. But, by protecting record timestamps from "going backwards," replication can do what is possible to ensure that the tables on replicated systems stay synchronized.

Conflict reporting

TimesTen conflict checking may be configured to report conflicts to a human-readable plain text file, or to an XML file for use by user applications. This section includes the topics:

Reporting conflicts to a text file
Reporting conflicts to an XML file
Reporting uniqueness conflicts
Reporting update conflicts
Reporting DELETE/UPDATE conflicts
Suspending and resuming the reporting of conflicts

Reporting conflicts to a text file

To configure replication to report conflicts to a human-readable text file (the default), use:

CHECK CONFLICTS BY ROW TIMESTAMP
  COLUMN ColumnName
  ...
  REPORT TO 'FileName' FORMAT STANDARD

An entry is added to the report file FileName that describes each conflict. The phrase FORMAT STANDARD is optional and may be omitted, as the standard report format is the default.

Each failed operation logged in the report consists of an entry that starts with a header, followed by information specific to the conflicting operation. Each entry is separated by a number of blank lines in the report.

The header contains:

The time the conflict was discovered.
The data stores that sent and received the conflicting update.
The table in which the conflict occurred.

The header has the following format:

Conflict detected at time on date
Datastore : subscriber_datastore
Transmitting name : master_datastore
Table : username.tablename

For example:

Conflict detected at 20:08:37 on 05-17-2004
Datastore : /tmp/subscriberds
Transmitting name : MASTERDS
Table : USER1.T1

Following the header is the information specific to the conflict. Data values are shown in ASCII format. Binary data is translated into hexadecimal before display, and floating-point values are shown with appropriate precision and scale.

For further description of the conflict report file, see "Reporting uniqueness conflicts", "Reporting update conflicts" and "Reporting DELETE/UPDATE conflicts".

Reporting conflicts to an XML file

To configure replication to report conflicts to an XML file, use:

CHECK CONFLICTS BY ROW TIMESTAMP
  COLUMN ColumnName
  ...
  REPORT TO 'FileName' FORMAT XML

Replication uses the base file name FileName to create two files. FileName.xml is a header file that contains the XML Document Type Definition for the conflict report structure, as well as the root element, defined as <ttrepconflictreport>. Inside the root element is an XML directive to include the file FileName.include, and it is to this file that all conflicts are written. Each conflict is written as a single element of type <conflict>.

For further description of the conflict report file XML elements, see Chapter 12, "XML Document Type Definition for the Conflict Report File".

Note:

When performing log maintenance on an XML conflict report file, only the file FileName.include should be truncated or moved. For conflict reporting to continue to function correctly, the file FileName.xml should be left untouched.

Reporting uniqueness conflicts

A uniqueness conflict record is issued when a replicated INSERT fails because of a conflict.

A uniqueness conflict record in the report file contains:

The timestamp and values for the existing tuple, which is the tuple that the conflicting tuple is in conflict with.
The timestamp and values for the conflicting insert tuple, which is the tuple of the insert that failed.
The key column values used to identify the record.
The action that was taken when the conflict was detected (discard the single row insert or the entire transaction)

Note:
If the transaction was discarded, the contents of the entire transaction are logged in the report file.

The format of a uniqueness conflict record is:

Conflicting insert tuple timestamp : <timestamp in binary format>
Existing tuple timestamp : <timestamp in binary format>
The existing tuple :
<<column value> [,<column value>. ..]>
The conflicting tuple :
<<column value> [,<column value> ...]>
The key columns for the tuple:
<<key column name> : <key column value>>
Transaction containing this insert skipped
Failed transaction:
Insert into table <user>.<table> <<columnvalue> [,<columnvalue>...]>
End of failed transaction

Example 11-3 shows the output from a uniqueness conflict on the row identified by the primary key value, '2'. The older insert replicated from subscriberds conflicts with the newer insert in masterds, so the replicated insert is discarded.

Example 11-3

Conflict detected at 13:36:00 on 03-25-2002
Datastore : /tmp/masterds
Transmitting name : SUBSCRIBERDS
Table : TAB
Conflicting insert tuple timestamp : 3C9F983D00031128
Existing tuple timestamp : 3C9F983E000251C0
The existing tuple :
< 2, 2, 3C9F983E000251C0>
The conflicting tuple :
< 2, 100, 3C9F983D00031128>
The key columns for the tuple:
<COL1 : 2>
Transaction containing this insert skipped
Failed transaction:
Insert into table TAB < 2, 100, 3C9F983D00031128>
End of failed transaction

Reporting update conflicts

An update conflict record is issued when a replicated UPDATE fails because of a conflict. This record reports:

The timestamp and values for the existing tuple, which is the tuple that the conflicting tuple is in conflict with.
The timestamp and values for the conflicting update tuple, which is the tuple of the update that failed.
The old values, which are the original values of the conflicting tuple before the failed update.
The key column values used to identify the record.
The action that was taken when the conflict was detected (discard the single row update or the entire transaction).

Note:
If the transaction was discarded, the contents of the entire transaction are logged in the report file.

The format of an update conflict record is:

Conflicting update tuple timestamp : <timestamp in binary format>
Existing tuple timestamp : <timestamp in binary format>
The existing tuple :
<<column value> [,<column value>. ..]>
The conflicting update tuple :
TSTAMP :<timestamp> :<<column value> [,<column value>. ..]>
The old values in the conflicting update:
TSTAMP :<timestamp> :<<column value> [,<column value>. ..]>
The key columns for the tuple:
<<key column name> : <key column value>>
Transaction containing this update skipped
Failed transaction:
Update table <user>.<table> with keys:
<<key column name> : <key column value>>
New tuple value:
<TSTAMP :<timestamp> :<<column value> [,<column value>. ..]>
End of failed transaction

Example 11-4 shows the output from an update conflict on the col2 value in the row identified by the primary key value, '6'. The older update replicated from the masterds data store conflicts with the newer update in subscriberds, so the replicated update is discarded.

Example 11-4 Output from an update conflict

Conflict detected at 15:03:18 on 03-25-2002
Datastore : /tmp/subscriberds
Transmitting name : MASTERDS
Table : TAB
Conflicting update tuple timestamp : 3C9FACB6000612B0
Existing tuple timestamp : 3C9FACB600085CA0
The existing tuple :
< 6, 99, 3C9FACB600085CA0>
The conflicting update tuple :
<TSTAMP :3C9FACB6000612B0, COL2 : 50>
The old values in the conflicting update:
<TSTAMP :3C9FAC85000E01F0, COL2 : 2>
The key columns for the tuple:
<COL1 : 6>
Transaction containing this update skipped
Failed transaction:
Update table TAB with keys:
<COL1 : 6>
New tuple value: <TSTAMP :3C9FACB6000612B0, COL2 : 50>
End of failed transaction

Reporting DELETE/UPDATE conflicts

A delete/update conflict record is issued when an update is attempted on a row that has more recently been deleted. This record reports:

The timestamp and values for the conflicting update tuple or conflicting delete tuple, whichever tuple failed.
If the delete tuple failed, the report also includes the timestamp and values for the existing tuple, which is the surviving update tuple with which the delete tuple was in conflict.
The key column values used to identify the record.
The action that was taken when the conflict was detected (discard the single row update or the entire transaction).

Note:
If the transaction was discarded, the contents of the entire transaction are logged in the report file. TimesTen cannot detect DELETE/INSERT conflicts.

The format of a record that indicates a delete conflict with a failed update is:

Conflicting update tuple timestamp : <timestamp in binary format>
The conflicting update tuple :
TSTAMP :<timestamp> :<<column value> [,<column value>. ..]>
This transaction skipped
The tuple does not exist
Transaction containing this update skipped
Update table <user>.<table> with keys:
<<key column name> : <key column value>>
New tuple value:
<TSTAMP :<timestamp> :<<column value> [,<column value>. ..]>
End of failed transaction

Example 11-5 shows the output from a delete/update conflict caused by an update on a row that has more recently been deleted. Because there is no row to update, the update from SUBSCRIBERDS is discarded.

Example 11-5 Output from a delete/update conflict: delete is more recent

Conflict detected at 15:27:05 on 03-25-2002
Datastore : /tmp/masterds
Transmitting name : SUBSCRIBERDS
Table : TAB
Conflicting update tuple timestamp : 3C9FB2460000AFC8
The conflicting update tuple :
<TSTAMP :3C9FB2460000AFC8, COL2 : 99>
The tuple does not exist
Transaction containing this update skipped
Failed transaction:
Update table TAB with keys:
<COL1 : 2>
New tuple value: <TSTAMP :3C9FB2460000AFC8,
COL2 : 99>
End of failed transaction

The format of a record that indicates an update conflict with a failed delete is:

Conflicting binary delete tuple timestamp : <timestamp in binary format>
Existing binary tuple timestamp : <timestamp in binary format>
The existing tuple :
<<column value> [,<column value>. ..]>
The key columns for the tuple:
<<key column name> : <key column value>>
Transaction containing this delete skipped
Failed transaction:
Delete table <user>.<table> with keys:
<<key column name> : <key column value>>
End of failed transaction

Example 11-6 shows the output from a delete/update conflict caused by a delete on a row that has more recently been updated. Because the row was updated more recently than the delete, the delete from masterds is discarded.

Example 11-6 Output from a delete/update conflict: update is more recent

Conflict detected at 15:27:20 on 03-25-2002
Datastore : /tmp/subscriberds
Transmitting name : MASTERDS
Table : TAB
Conflicting binary delete tuple timestamp : 3C9FB258000708C8
Existing binary tuple timestamp : 3C9FB25800086858
The existing tuple :
< 147, 99, 3C9FB25800086858>
The key columns for the tuple:
<COL1 : 147>
Transaction containing this delete skipped
Failed transaction:
Delete table TAB with keys:
<COL1 : 147>

Suspending and resuming the reporting of conflicts

Provided your applications are well-behaved, replication usually encounters and reports only sporadic conflicts. However, it is sometimes possible under heavy load to trigger a flurry of conflicts in a short amount of time, particularly when applications are in development and such errors are expected. This can potentially have a negative impact on the performance of the host machine because of excessive writes to the conflict report file and the large number of SNMP traps that can be generated.

To avoid overwhelming a host with replication conflicts, you may configure replication to suspend conflict reporting when the number of conflicts per second has exceeded a user-specified threshold. Conflict reporting may also be configured to resume once the conflicts per second have fallen below a user-specified threshold.

Conflict reporting suspension and resumption can be detected by an application by catching the SNMP traps ttRepConflictReportStoppingTrap and ttRepConflictReportStartingTrap, respectively. See "Diagnostics through SNMP Traps" in Oracle TimesTen In-Memory Database Error Messages and SNMP Traps for more information.

To configure conflict reporting to be suspended and resumed based on the number of conflicts per second, use the CONFLICT REPORTING SUSPEND AT and CONFLICT REPORTING RESUME AT attributes for the STORE clause of a replication scheme. Example 11-7 demonstrates the configuration of a replication schemes where conflict reporting ceases when the number of conflicts exceeds 20 per second, and conflict reporting resumes when the number of conflicts drops below 10 per second.

Note:

If the replication agent is stopped while conflict reporting is suspended, conflict reporting is enabled when the replication agent is restarted. The SNMP trap ttRepConflictReportingStartingTrap is not sent if this occurs. This means that an application that monitors the conflict report suspension traps must also monitor the traps for replication agent stopping and starting.

Example 11-7 Configuring conflict reporting thresholds

CREATE REPLICATION r1
ELEMENT elem_accounts_1 TABLE accounts
      CHECK CONFLICTS BY ROW TIMESTAMP
        COLUMN tstamp
        UPDATE BY SYSTEM
        ON EXCEPTION ROLLBACK WORK
        REPORT TO 'conflicts' FORMAT XML
  MASTER westds ON "westcoast"
  SUBSCRIBER eastds ON "eastcoast"
ELEMENT elem_accounts_2 TABLE accounts
      CHECK CONFLICTS BY ROW TIMESTAMP
        COLUMN tstamp
        UPDATE BY SYSTEM
        ON EXCEPTION ROLLBACK WORK
        REPORT TO 'conflicts' FORMAT XML
  MASTER eastds ON "eastcoast"
  SUBSCRIBER westds ON "westcoast"
STORE westds ON "westcoast"
  CONFLICT REPORTING SUSPEND AT 20
  CONFLICT REPORTING RESUME AT 10
STORE eastds ON "eastcoast"
  CONFLICT REPORTING SUSPEND AT 20
  CONFLICT REPORTING RESUME AT 10;

Managing data store failover and recovery

As discussed in "Designing a highly available system", a fundamental element in the design of a highly available system is the ability to quickly recover from a failure. Failures may be related to:

Hardware problems:

System failure
Network failure

Software problems:

Operating system failure
Application failure
Data store failure
Operator error

Your replicated system must employ a "cluster manager" or custom software to detect such failures and, in the event of a failure involving a master data store, redirect the user load to one of its subscribers. TimesTen does not provide a cluster manager or make any assumptions about how they operate, so the focus of this discussion is on the TimesTen mechanisms that an application or cluster manager can use to recover from failures.

Unless the replication scheme is configured to use the return twosafe service, TimesTen replicates updates only after the original transaction commits to the master data store. If a subscriber data store is inoperable or communication to a subscriber data store fails, updates at the master are not impeded. During outages at subscriber systems, updates intended for the subscriber are saved in the TimesTen transaction log.

Note:

The procedures described in this section require that you have the ADMIN privilege.

General failover and recovery procedures

The procedures for managing failover and recovery depend primarily on:

Your replication scheme.
Whether the failure occurred on a master or subscriber data store.
Whether the threshold for the transaction log on the master is exhausted before the problem is resolved and the data stores reconnected.

Subscriber failures

If your replication scheme is configured for default asynchronous replication, should a subscriber data store become inoperable or communication to a subscriber data store fail, updates at the master are not impeded and the cluster manager does not have to take any immediate action.

Note:

If the failed subscriber is configured to use a return service, you must first disable return service blocking, as described in "Managing return service timeout errors and replication state changes"6.

During outages at subscriber systems, updates intended for the subscriber are saved in the transaction log on the master. If the subscriber agent reestablishes communication with its master before the master reaches its FAILTHRESHOLD, the updates held in the log are automatically transferred to the subscriber and no further action is required. (See "Setting the log failure threshold" for details on how to establish the FAILTHRESHOLD value for the master data store.)

If the FAILTHRESHOLD is exceeded, the master sets the subscriber to the Failed state and it must be recovered, as described in "Recovering a failed data store". Any application that connects to the failed subscriber receives a tt_ErrReplicationInvalid (8025) warning indicating that the data store has been marked Failed by a replication peer.

Applications can use the ODBC SQLGetInfo function to check if the subscriber data store it is connected to has been set to the Failed state. The SQLGetInfo function includes a TimesTen-specific infotype, TT_REPLICATION_INVALID, that returns a 32-bit integer value of '1' if the data store is failed, or '0' if not failed. Since the infotype TT_REPLICATION_INVALID is specific to TimesTen, all applications using it need to include the timesten.h file in addition to the other ODBC include files.

Example 11-8 Checking whether a data store has been set to the Failed state

Check if the data store identified by the hdbc handle has been set to the Failed state.

SQLINTEGER retStatus;

SQLGetInfo(hdbc, TT_REPLICATION_INVALID,
          (PTR)&retStatus, NULL, NULL);

Master failures

The cluster manager plays a more central role if a failure involves the master data store. Should a master data store fail, the cluster manager must detect this event and redirect the user load to one of its surviving data stores. This surviving subscriber then becomes the master, which continues to accept transactions and replicates them to the other surviving subscriber data stores. If the failed master and surviving subscriber are configured in a bidirectional manner, transferring the user load from a failed master to a subscriber does not require that you make any changes to your replication scheme. However, when using unidirectional replication or complex schemes, such as those involving propagators, you may have to issue one or more ALTER REPLICATION statements to reconfigure the surviving subscriber as the "new master" in your scheme. See "Replacing a master data store" for an example.

When the problem is resolved, if you are not using the hot-standby configuration or the active standby pair described in "Automatic catch-up of a failed master data store", you must recover the master data store as described in "Recovering a failed data store".

After the data store is back online, the cluster manager can either transfer the user load back to the original master or reestablish it as a subscriber for the "acting master." See "Failover and recovery" on page 40 for more information.

Automatic catch-up of a failed master data store

The master catch-up feature automatically restores a failed master data store from a subscriber data store without the need to invoke the ttRepAdmin -duplicate operation described in "Recovering a failed data store".

The master catch-up feature needs no configuration, but it can be used only in the following types of configurations:

A single master replicated in a bidirectional, hot-standby manner to a single subscriber
An active standby pair in which the active master data store is replicated to the standby data store which then propagates changes to up to 127 read-only subscribers

In addition, the following must be true:

The ELEMENT type is DATASTORE.
TRANSMIT NONDURABLE or RETURN TWOSAFE must be enabled. TRANSMIT NONDURABLE is optional for asynchronous and return receipt transactions.
All replicated transactions must be committed non-durably (they must be transmitted to the other master data store before they are committed on the local one). For example, if the replication scheme is configured with RETURN TWOSAFE BY REQUEST and any transaction is committed without first enabling RETURN TWOSAFE, master catch-up may not occur after a failure of the master.

When the master replication agent is restarted after a crash or invalidation, any lost transactions that originated on the master are automatically reapplied from the subscriber to the master. No connections are allowed to the master store until it has completely caught up with the subscriber. Applications attempting to connect to a data store during the catch-up phase receive an error that indicates a catch-up is in progress. The only exception is if you connect to a data store with the ForceConnect attribute set in the DSN.

When the catch-up phase is complete, your application can connect to the data store. An SNMP trap and message to the system log indicate the completion of the catch-up phase.

If one of the stores is invalidated or crashes during the catch-up process, the catch-up phase is resumed when the store comes back up.

Master/subscriber failures

You can distribute the workload over multiple bidirectionally replicated data stores, each of which serves as both master and subscriber. When recovering a master/subscriber data store, the log on the failed data store may present problems when you restart replication. See "Bidirectional general workload scheme".

If a data store in a distributed workload scheme fails and work is shifted to a surviving data store, the information in the surviving data store becomes more current than that in the failed data store. If replication is restarted at the failed system before the FAILTHRESHOLD has been reached on the surviving data store, then both data stores attempt to update one another with the contents of their transaction logs. In this case, the older updates in the transaction log on the failed data store may overwrite more recent data on the surviving system.

There are two ways to recover in such a situation:

If the timestamp conflict resolution rules described in "Replication conflict detection and resolution" are sufficient to guarantee consistency for your application, then you can restart the failed system and allow the updates from the failed data store to propagate to the surviving data store. The conflict resolution rules prevent more recent updates from being overwritten.
Re-create the failed data store, as described in "Recovering a failed data store".

Note:
If the data store must be re-created, the updates in the log on the failed data store that were not received by the surviving data store cannot be identified or restored. In the case of several surviving data stores, you must select which of the surviving data stores is to be used to re-create the failed data store. It is possible that at the time the failed data store is re-created, that the selected surviving data store may not have received all updates from the other surviving data stores. This results in diverging data stores. The only way to prevent this situation is to re-create the other surviving data stores from the selected surviving data store.

Network failures

In the event of a temporary network failure, you need not perform any specific action to continue replication. The replication agents that were in communication attempt to reconnect every few seconds. Should the agents reconnect before the master data store runs out of log space, the replication protocol makes sure they neither miss nor repeat any replication updates. If the network is unavailable for a longer period and the FAILTHRESHOLD has been exceeded for the master log, you need to recover the subscriber as described in "Recovering a failed data store".

Failures involving sequences

After a link failure, if replication is allowed to recover by replaying queued logs, you do not need to take any action.

However, if the failed node was down for a significant amount of time, you must use the ttRepAdmin -duplicate command to repopulate the data store on the failed node with transactions from the surviving node, as sequences are not rolled back during failure recovery. In this case, the ttRepAdmin -duplicate command copies the sequence definitions from one node to the other.

Recovering a failed data store

If a restarted data store cannot be recovered from its master's transaction log so that it is consistent with the other data stores in the replicated system, you must re-create the data store from one of its replication peers. If your data stores are configured in a hot-standby replication scheme, as described in "Automatic catch-up of a failed master data store", a failed master data store is automatically brought up to date from the subscriber. Data stores configured with other types of replication schemes must be restored using command line utilities or programmatically using the TimesTen Utility C functions, as described below.

Note:

It is not necessary to re-create the DSN for the failed data store.

In the event of a subscriber failure, if any tables are configured with a return service, commits on those tables in the master data store are blocked until the return service timeout period expires. To avoid this, you can establish a return service failure and recovery policy in your replication scheme, as described in"Managing return service timeout errors and replication state changes". If you are using the RETURN RECEIPT service, an alternative is to use ALTER REPLICATION and set the NO RETURN attribute to disable return receipt until the subscriber is restored and caught up. Then, you can submit another ALTER REPLICATION to re-establish RETURN RECEIPT.

From the command line

If the data stores are fully replicated, you can use ttDestroy to remove the failed data store from memory and ttRepAdmin -duplicate to re-create it from a surviving data store. If the data store contains any cache groups, you must also use the -keepCG option of ttRepAdmin. See "Duplicating a data store".

Example 11-9 Recovering a failed data store

To recover a failed data store, subscriberds, from a master, named masterds on host system1, enter:

> ttDestroy /tmp/subscriberds

> ttRepAdmin -dsn subscriberds -duplicate -from masterds -host "system1" -uid ttuser

You will be prompted for the password of ttuser.

Note:

ttRepAdmin -duplicate is only supported between identical and patch TimesTen releases (the major and minor release numbers must be the same).

After re-creating the data store with ttRepAdmin -duplicate, the first connection to the data store reloads it into memory. To improve performance when duplicating large data stores, you can avoid the reload step by using the ttRepAdmin -ramLoad option to keep the data store in memory after the duplicate operation.

Example 11-10 Keeping a data store in memory when recovering it

To recover a failed data store, subscriberds, from a master, named masterds on host system1, and to keep the data store in memory and restart replication after the duplicate operation, enter:

> ttDestroy /tmp/subscriberds

> ttRepAdmin -dsn subscriberds -duplicate -ramLoad -from masterds -host "system1" -uid ttuser -setMasterRepStart

You will be prompted for the password of ttuser.

Note:

After duplicating a data store with the ttRepAdmin -duplicate -ramLoad options, the RAM Policy for the data store is manual until explicitly reset by ttAdmin -ramPolicy or the ttRamPolicy function.

From a program

You can use the C functions provided in the TimesTen Utility library to programmatically recover a failed data store.

If the data stores are fully replicated, you can use ttDestroyDataStore function to remove the failed data store and the ttRepDuplicateEx function to re-create it from a surviving data store.

Example 11-11 Recovering and starting a failed data store

To recover and start a failed data store, named subscriberds on host system2, from a master, named masterds on host system1, enter:

int          rc;
ttUtilHandle utilHandle;
ttRepDuplicateExArg arg;
memset( &arg, 0, sizeof( arg ) );
arg.size = sizeof( ttRepDuplicateExArg );
arg.flags = TT_REPDUP_REPSTART | TT_REPDUP_RAMLOAD;
arg.uid=ttuser;
arg.pwd=ttuser;
arg.localHost = "system2";
rc = ttDestroyDataStore( utilHandle, "subscriberds", 30 );
rc = ttRepDuplicateEx( utilHandle, "DSN=subscriberds",
                      "masterds", "system1", &arg );

In this example, the timeout for the ttDestroyDataStore operation is 30 seconds. The last parameter of the ttRepDuplicateEx function is an argument structure containing two flags: TT_REPDUP_RESTART to set the subscriberds data store to the Start state after the duplicate operation is completed, and TT_REPDUP_RAMLOAD to set the RAM Policy to manual and keep the data store in memory.

Note:

When the TT_REPDUP_RAMLOAD flags is used with ttRepDuplicateEx, the RAM policy for the duplicate data store is manual until explicitly reset by the ttRamPolicy function or ttAdmin -ramPolicy.

See "TimesTen Utility API" in Oracle TimesTen In-Memory Database C Developer's Guide for the complete list of the functions provided in the TimesTen C Language Utility Library.

Recovering NONDURABLE data stores

If your data store is configured with the TRANSMIT NONDURABLE option in a hot-standby configuration, as described in "Automatic catch-up of a failed master data store", you do not need to take any action to recover a failed master data store.

For other types of configurations, if the master data store configured with the TRANSMIT NONDURABLE option fails, you must use ttRepAdmin-duplicate or ttRepDuplicateEx to re-create the master data store from the most current subscriber data store. If your application attempts to reconnect to the master store without first performing the duplicate operation, the replication agent recovers the data store, but any attempt to connect results in an error that advises you to perform the 'duplicate'. To avoid this error, your application must reconnect with the connection attribute, ForceConnect set to 1.

Writing a failure recovery script

Upon detecting a failure, the cluster manager should invoke a script that effectively executes the procedure shown by the pseudocode in Example 11-12.

Example 11-12 Failure recovery pseudocode

Detect problem {
       if (Master == unavailable) {
          FailedDataStore = Master
          FailedDSN = Master_DSN
          SurvivorDataStore = Subscriber
          switch users to SurvivorDataStore
      }
else {
          FailedDataStore = Subscriber
          FailedDSN = Subscriber_DSN
          SurvivorDataStore = Master
      }
}
Fix problem....
If (Problem resolved) {
       Get state for FailedDataStore
       if (state == "failed") {
         ttDestroy FailedDataStore
         ttRepAdmin -dsn FailedDSN -duplicate
                 -from SurvivorDataStore -host SurvivorHost
                 -setMasterRepStart
                 -uid ttuser
                 -pwd ttuser
      }
      else {
         ttAdmin -repStart FailedDSN
      }
      while (backlog != 0) {
         wait
      }
}

Switch users back to Master.

This applies to either the master or subscriber data stores. If the master fails, you may lose some transactions.