5 Troubleshooting AWT Cache Groups

Unable to start or stop replication agent

This section describes what to check if you are unable to start or stop a replication agent.

Replication does not work

If you are unable to get replication working, the problem may be one or more of the following:

Using SNMP traps for notification of replication events

TimesTen can send SNMP traps for certain replication events to enable network management software to take immediate action. TimesTen can send the following SNMP traps:

• ttRepAgentExitingTrap

• ttRepAgentDiedTrap

• ttRepAgentStartingTrap

These traps are described in "Diagnostics through SNMP Traps" in the Oracle TimesTen In-Memory Database Error Messages and SNMP Traps.

Monitoring AWT performance

You can monitor the performance of asynchronous writethrough (AWT) cache groups to determine how much time is spent performing tasks in the AWT workflow. Use the ttCacheAwtMonitorConfig built-in procedure to enable monitoring.

For example, enable monitoring and set the sampling frequency to 16. A sampling factor of 16 is recommended for accuracy and performance.

Command> CALL ttCacheAwtMonitorConfig('ON',16);
< ON, 16 >
1 row found.

Use the ttRepAdmin utility with the -awtmoninfo and -showstatus commands to display the monitoring results. The AWT monitoring statistics include:

• TimesTen processing time: The total number of milliseconds spent in processing AWT transaction data since monitoring was enabled.

• Oracle bookmark management time: The total number of milliseconds spent in managing AWT metadata on Oracle since monitoring was enabled.

• Oracle execute time: The total number of milliseconds spent in OCI preparation, binding and execution for AWT SQL operations since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.

• Oracle commit time: The total number of milliseconds spent in committing AWT updates on Oracle since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.

• Time since monitoring was started

• Total number of TimesTen row operations: The total number of rows updated in AWT cache groups since monitoring was enabled.

• Total number of TimesTen transactions: The total number of transactions in AWT cache groups since monitoring was enabled.

• Total number of flushes to Oracle: The total number of times that TimesTen data has been sent to Oracle.

The output also includes the percentage of time spent on TimesTen processing, Oracle bookmark management, Oracle execution and Oracle commits.

For example:

ttRepAdmin -showstatus -awtmoninfo myDSN

[other -showstatus output]
...
AWT Monitoring statistics
-------------------------
TimesTen processing time : 0.689000 millisecs (0.164307 %)
   Oracle bookmark management time : 3.229000 millisecs (0.770027%)
   Oracle execute time : 342.908000 millisecs (81.774043 %)
   Oracle commit time : 72.450000 millisecs (17.277315 %)
   Time since monitoring was started: 8528.641000 millisecs
   Cache-connect Operational Stats :
      Total Number of TimesTen row operations : 2
      Total Number of TimesTen transactions : 2
      Total Number of flushes to Oracle : 2

Possible causes of poor AWT performance

This section addresses issues that may degrade AWT performance.

Permanent Oracle errors reported by TimesTen

Insert, update, or delete errors that occur while applying changes to Oracle are saved in an error file located in the data store directory with the following name:

DatastoreName.awterr

Errors reported to this file are permanent errors. TimesTen does not retry the transaction. The errors may be reported in the AWT error file long after the commit to TimesTen occurs.

The format of the messages in the AWT error file is similar to those generated for conflict and transaction errors in replication, as shown in Example 5-1. Oracle error messages are also reported in the support log and the user log.

Error occurred 14:48:55 on 03-22-2007
Datastore: c:\temp\cgDSN
Oracle Id: system1
Transmitting name: cgDSN
Error message: 
  TT5210: Oracle unique constraint violation error in OCIStmtExecute(): ORA-00001: unique constraint (GUSER.SYS_C00357240) violated rc = -1 -- file "bdbTblH.c", lineno 1205, procedure "ttBDbStmtForce()"
  TT5025: Commit failure in Oracle. Transaction must be rolled back in TimesTen. -- file "bdbConnect.c", lineno 885, procedure "ttBDbXact()"

Operation that caused the error:
Insert into table TESTUSER.T1 <9,1000>

Failed transaction:
Insert into table TESTUSER.T1 <9, 1000>
End of failed transaction

May 04 18:12:36 HOST1 TimesTen Replication 7.0[2136]: 
[Err ] DEFAULT:meta.c(639): 
TT16062: Failed to compile command:
select p.commit_timestamp, p.commit_seqnum, p.protocol from owner1.TT_03_REPPEERS p where p.replication_name = :rname and p.replication_owner = :rowner and p.tt_store_id   = :oid and p.subscriber_id = :sid

May 04 18:12:36 HOST1 TimesTen Replication 7.0[2136]: 
[Err ] DEFAULT:meta.c(639): 
TT5221: TT5221: Oracle syntax error in OCIStmtExecute(): 
ORA-00942: table or view does not exist rc = -1 -- file "bdbStmt.c", lineno 1041, procedure "getOraOutTypesNLengths()"

1. Stop the replication agent.

2. Drop and re-create the cache group.

3. Restart the replication agent.

Transient Oracle errors reported by TimesTen

The support log for data stores with AWT cache groups may contain Oracle errors if the replication agent encounters a problem on the Oracle database. If the replication agent encounters one of these errors, AWT rolls back the transaction and retries it. If the support log becomes full, the oldest messages are deleted and replaced by new messages.

The Oracle errors in the support log are considered transient because AWT retries the transaction.

Some transient errors indicate an underlying problem on the Oracle database must be solved before AWT operations can continue. For example:

ORA-01536: space quota exceeded for tablespace
ORA-01034: ORACLE not available

After the underlying problem has been fixed, AWT retries the operation.

For more information about the Oracle errors, see Oracle Database Error Messages for the Oracle release you are using.

The following Oracle errors are transient:

ORA-00018: maximum number of sessions exceeded
ORA-00019: maximum number of session licenses exceeded
ORA-00020: maximum number of processes (%s) exceeded
ORA-00025: failed to allocate %s
ORA-00028: your session has been killed
ORA-00038: Cannot create session: server group belongs to another user
ORA-00051: timeout occurred while waiting for a resource
ORA-00052: maximum number of enqueue resources (%s) exceeded
ORA-00053: maximum number of enqueues exceeded
ORA-00054: resource busy and acquire with NOWAIT specified
ORA-00055: maximum number of DML locks exceeded
ORA-00057: maximum number of temporary table locks exceeded
ORA-00058: DB_BLOCK_SIZE must be %s to mount this database (not %s)
ORA-00059: maximum number of DB_FILES exceeded
ORA-00060: deadlock detected while waiting for resource
ORA-00063: maximum number of LOG_FILES exceeded
ORA-00064: object is too large to allocate on this O/S (%s,%s)
ORA-00099: timed out while waiting for resource, potential PDML deadlock
ORA-00104: deadlock detected; all public servers blocked waiting for resources
ORA-00107: failed to connect to ORACLE listener process
ORA-00115: connection refused; dispatcher connection table is full
ORA-00125: connection refused; invalid presentation
ORA-00126: connection refused; invalid duplicity
ORA-00284: recovery session still in progress
ORA-00370: potential deadlock during kcbchange operation
ORA-00371: not enough shared pool memory
ORA-00376: file %s cannot be read at this time
ORA-00379: no free buffers available in buffer pool %s for block size %sK
ORA-00384: Insufficient memory to grow cache
ORA-00568: Maximum number of interrupt handlers exceeded
ORA-00579: osndnt: server received malformed connection request
ORA-00600: internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s], [%s]
ORA-00603: ORACLE server session terminated by fatal error
ORA-01000: maximum open cursors exceeded
ORA-01012: not logged on
ORA-01014: ORACLE shutdown in progress
ORA-01019: unable to allocate memory in the user side
ORA-01031: insufficient privileges
ORA-01033: ORACLE initialization or shutdown in progress
ORA-01034: ORACLE not available
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege
ORA-01037: maximum cursor memory exceeded
ORA-01046: cannot acquire space to extend context area
ORA-01073: fatal connection error: unrecognized call type
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-01090: shutdown in progress - connection is not permitted
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-01094: ALTER DATABASE CLOSE in progress. Connections not permitted
ORA-01109: database not open
ORA-01147: SYSTEM tablespace file %s is offline
ORA-01154: database busy. Open, close, mount, and dismount not allowed now
ORA-01155: the database is being opened, closed, mounted or dismounted
ORA-01219: database not open: queries allowed on fixed tables/views only
ORA-01237: cannot extend datafile %s
ORA-01456: may not perform insert/delete/update operation inside a READ ONLY transaction
ORA-01536: space quota exceeded for tablespace '%s'
ORA-01539: tablespace '%s' is not online
ORA-01542: tablespace '%s' is offline, cannot allocate space in it
ORA-01562: failed to extend rollback segment number %s
ORA-01573: shutting down instance, no further change allowed
ORA-01628: max # extents (%s) reached for rollback segment %s
ORA-01629: max # extents (%s) reached saving undo for tablespace %s
ORA-01630: max # extents (%s) reached in temp segment in tablespace %s
ORA-01631: max # extents (%s) reached in table %s.%s
ORA-01632: max # extents (%s) reached in index %s.%s
ORA-01650: unable to extend rollback segment %s by %s in tablespace %s
ORA-01651: unable to extend save undo segment by %s for tablespace %s
ORA-01652: unable to extend temp segment by %s in tablespace %s
ORA-01653: unable to extend table %s.%s by %s in tablespace %s
ORA-01654: unable to extend index %s.%s by %s in tablespace %s
ORA-01655: unable to extend cluster %s.%s by %s in tablespace %s
ORA-01656: max # extents (%s) reached in cluster %s.%s
ORA-01658: unable to create INITIAL extent for segment in tablespace %s
ORA-01659: unable to allocate MINEXTENTS beyond %s in tablespace %s
ORA-01680: unable to extend LOB segment by %s in tablespace %s
ORA-01681: max # extents (%s) reached in LOB segment in tablespace %s
ORA-01683: unable to extend index %s.%s partition %s by %s in tablespace %s
ORA-01684: max # extents (%s) reached in table %s.%s partition %s
ORA-01685: max # extents (%s) reached in index %s.%s partition %s
ORA-01686: max # files (%s) reached for the tablespace %s
ORA-01688: unable to extend table %s.%s partition %s by %s in tablespace %s
ORA-01691: unable to extend lob segment %s.%s by %s in tablespace %s
ORA-01692: unable to extend lob segment %s.%s partition %s by %s in tablespace %s
ORA-01693: max # extents (%s) reached in lob segment %s.%s
ORA-01694: max # extents (%s) reached in lob segment %s.%s partition %s
ORA-03113: end-of-file on communication channel
ORA-03114: not connected to ORACLE
ORA-03134: Connections to this server version are no longer supported.
ORA-03135: connection lost contact
ORA-03136: inbound connection timed out
ORA-03232: unable to allocate an extent of %s blocks from tablespace %s
ORA-03233: unable to extend table %s.%s subpartition %s by %s in tablespace %s
ORA-03234: unable to extend index %s.%s subpartition %s by %s in tablespace %s
ORA-03235: max # extents (%s) reached in table %s.%s subpartition %s
ORA-03236: max # extents (%s) reached in index %s.%s subpartition %s
ORA-03237: Initial Extent of specified size cannot be allocated
ORA-03238: unable to extend LOB segment %s.%s subpartition %s by %s in tablespace %s
ORA-03239: maxextents (%s) reached in LOB segment %s.%s subpartition %s
ORA-04020: deadlock detected while trying to lock object %s%s%s%s%s
ORA-06019: NETASY: invalid login (connect) string
ORA-06021: NETASY: connect failed
ORA-06030: NETDNT: connect failed, unrecognized node name
ORA-06031: NETDNT: connect failed, unrecognized object name
ORA-06032: NETDNT: connect failed, access control data rejected
ORA-06033: NETDNT: connect failed, partner rejected connection
ORA-06034: NETDNT: connect failed, partner exited unexpectedly
ORA-06035: NETDNT: connect failed, insufficient resources
ORA-06036: NETDNT: connect failed, no response from object
ORA-06037: NETDNT: connect failed, node unreachable
ORA-06039: NETDNT: connect failed
ORA-06040: NETDNT: invalid login (connect) string
ORA-06108: NETTCP: connect to host failed
ORA-06113: NETTCP: Too many connections
ORA-06114: NETTCP: SID lookup failure
ORA-06143: NETTCP: maximum connections exceeded
ORA-06315: IPA: Invalid connect string
ORA-06316: IPA: Invalid database SID
ORA-06317: IPA: Local maximum number of users exceeded
ORA-06318: IPA: Local maximum number of connections exceeded
ORA-06319: IPA: Remote maximum number of users exceeded
ORA-06320: IPA: Remote maximum number of connections exceeded
ORA-06404: NETCMN: invalid login (connect) string
ORA-06413: Connection not open.
ORA-10435: enable tracing of global enqueue service deadlock detetction
ORA-10626: specify timeout for online index rebuild to wait for DML
ORA-10906: Unable to extend segment after insert direct load
ORA-12150: TNS:unable to send data
ORA-12151: TNS:received bad packet type from network layer
ORA-12152: TNS:unable to send break message
ORA-12153: TNS:not connected
ORA-12154: TNS:could not resolve service name
ORA-12155: TNS:received bad datatype in NSWMARKER packet
ORA-12156: TNS:tried to reset line from incorrect state
ORA-12157: TNS:internal network communication error
ORA-12158: TNS:could not initialize parameter subsystem
ORA-12159: TNS:trace file not writeable
ORA-12160: TNS:internal error: Bad error number
ORA-12161: TNS:internal error: partial data received
ORA-12162: TNS:service name is incorrectly specified
ORA-12163: TNS:connect descriptor is too long
ORA-12166: TNS:Client can not connect to HO agent.
ORA-12168: TNS:Unable to contact Directory Server.
ORA-12169: TNS:Net service name given as connect identifier is too long
ORA-12170: TNS:Connect timeout occurred
ORA-12171: TNS:could not resolve connect identifier: %s
ORA-12196: TNS:received an error from TNS
ORA-12197: TNS:keyword-value resolution error
ORA-12198: TNS:could not find path to destination
ORA-12200: TNS:could not allocate memory
ORA-12201: TNS:encountered too small a connection buffer
ORA-12202: TNS:internal navigation error
ORA-12203: TNS:unable to connect to destination
ORA-12204: TNS:received data refused from an application
ORA-12205: TNS:could not get failed addresses
ORA-12206: TNS:received a TNS error during navigation
ORA-12207: TNS:unable to perform navigation
ORA-12208: TNS:could not find the TNSNAV.ORA file
ORA-12209: TNS:encountered uninitialized global
ORA-12210: TNS:error in finding Navigator data
ORA-12211: TNS:needs PREFERRED_CMANAGERS entry in TNSNAV.ORA
ORA-12212: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA
ORA-12213: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA
ORA-12214: TNS:missing local communities entry in TNSNAV.ORA
ORA-12215: TNS:poorly formed PREFERRED_NAVIGATORS Addresses in TNSNAV.ORA
ORA-12216: TNS:poorly formed PREFERRED_CMANAGERS addresses in TNSNAV.ORA
ORA-12217: TNS:could not contact PREFERRED_CMANAGERS in TNSNAV.ORA
ORA-12218: TNS:unacceptable network configuration data
ORA-12219: TNS:missing community name from address in ADDRESS_LIST
ORA-12221: TNS:illegal ADDRESS parameters
ORA-12222: TNS:no such protocol adapter
ORA-12223: TNS:internal limit restriction exceeded
ORA-12224: TNS:no listener
ORA-12225: TNS:destination host unreachable
ORA-12226: TNS:operating system resource quota exceeded
ORA-12227: TNS:syntax error
ORA-12228: TNS:protocol adapter not loadable
ORA-12229: TNS:Interchange has no more free connections
ORA-12230: TNS:Severe Network error ocurred in making this connection
ORA-12231: TNS:No connection possible to destination
ORA-12232: TNS:No path available to destination
ORA-12233: TNS:Failure to accept a connection
ORA-12235: TNS:Failure to redirect to destination
ORA-12236: TNS:protocol adapter not loaded
ORA-12316: syntax error in database link's connect string
ORA-12326: database %s is closing immediately;  no operations are permitted
ORA-12329: database %s is closed;  no operations are permitted
ORA-12500: TNS:listener failed to start a dedicated server process
ORA-12501: TNS:listener failed to spawn process
ORA-12502: TNS:listener received no CONNECT_DATA from client
ORA-12504: TNS:listener was not given the SID in CONNECT_DATA
ORA-12505: TNS:listener could not resolve SID given in connect descriptor
ORA-12506: TNS:listener was not given the ALIAS in CONNECT_DATA
ORA-12507: TNS:listener could not resolve ALIAS given
ORA-12508: TNS:listener could not resolve the COMMAND given
ORA-12509: TNS:listener failed to redirect client to service handler
ORA-12510: TNS:database temporarily lacks resources to handle the request
ORA-12511: TNS:service handler found but it is not accepting connections
ORA-12512: TNS:service handler found but it has not registered a redirect address
ORA-12513: TNS:service handler found but it has registered for a different protocol
ORA-12514: TNS:listener could not resolve SERVICE_NAME given in connect descriptor
ORA-12515: TNS:listener could not find a handler for this presentation
ORA-12516: TNS:listener could not find available handler with matching protocol stack
ORA-12517: TNS:listener could not find service handler supporting direct handoff
ORA-12518: TNS:listener could not hand off client connection
ORA-12519: TNS:no appropriate service handler found
ORA-12520: TNS:listener could not find available handler for requested type of server
ORA-12521: TNS:listener could not resolve INSTANCE_NAME given in connect descriptor
ORA-12522: TNS:listener could not find available instance with given INSTANCE_ROLE
ORA-12523: TNS:listener could not find instance appropriate for the client connection
ORA-12524: TNS:listener could not resolve HANDLER_NAME given in connect descriptor
ORA-12525: TNS:listener has not received client's request in time allowed
ORA-12526: TNS:listener: all appropriate instances are in restricted mode
ORA-12527: TNS:listener: all instances are in restricted mode or blocking new connections
ORA-12528: TNS:listener: all appropriate instances are blocking new connections
ORA-12529: TNS:connect request rejected based on current filtering rules
ORA-12531: TNS:cannot allocate memory
ORA-12532: TNS:invalid argument
ORA-12533: TNS:illegal ADDRESS parameters
ORA-12534: TNS:operation not supported
ORA-12535: TNS:operation timed out
ORA-12536: TNS:operation would block
ORA-12537: TNS:connection closed
ORA-12538: TNS:no such protocol adapter
ORA-12539: TNS:buffer over- or under-flow
ORA-12540: TNS:internal limit restriction exceeded
ORA-12541: TNS:no listener
ORA-12542: TNS:address already in use
ORA-12543: TNS:destination host unreachable
ORA-12544: TNS:contexts have different wait/test functions
ORA-12545: Connect failed because target host or object does not exist
ORA-12546: TNS:permission denied
ORA-12547: TNS:lost contact
ORA-12549: TNS:operating system resource quota exceeded
ORA-12550: TNS:syntax error
ORA-12551: TNS:missing keyword
ORA-12552: TNS:operation was interrupted
ORA-12554: TNS:current operation is still in progress
ORA-12555: TNS:permission denied
ORA-12556: TNS:no caller
ORA-12557: TNS:protocol adapter not loadable
ORA-12558: TNS:protocol adapter not loaded
ORA-12560: TNS:protocol adapter error
ORA-12561: TNS:unknown error
ORA-12562: TNS:bad global handle
ORA-12564: TNS:connection refused
ORA-12566: TNS:protocol error
ORA-12569: TNS:packet checksum failure
ORA-12570: TNS:packet reader failure
ORA-12571: TNS:packet writer failure
ORA-12574: TNS:redirection denied
ORA-12582: TNS:invalid operation
ORA-12583: TNS:no reader
ORA-12585: TNS:data truncation
ORA-12589: TNS:connection not bequeathable
ORA-12590: TNS:no I/O buffer
ORA-12591: TNS:event signal failure
ORA-12592: TNS:bad packet
ORA-12593: TNS:no registered connection
ORA-12595: TNS:no confirmation
ORA-12596: TNS:internal inconsistency
ORA-12600: TNS: string open failed
ORA-12602: TNS: Connection Pooling limit reached
ORA-12606: TNS: Application timeout occurred
ORA-12607: TNS: Connect timeout occurred
ORA-12608: TNS: Send timeout occurred
ORA-12609: TNS: Receive timeout occurred
ORA-12612: TNS:connection is busy
ORA-12615: TNS:preempt error
ORA-12623: TNS:operation is illegal in this state
ORA-12624: TNS:connection is already registered
ORA-12636: Packet send failed
ORA-12637: Packet receive failed
ORA-12829: Deadlock - itls occupied by siblings at block %s of file %s
ORA-12993: tablespace '%s' is offline, cannot drop column
ORA-14117: partition resides in offlined tablespace
ORA-14268: subpartition '%s' of the partition resides in offlined tablespace
ORA-16000: database open for read-only access
ORA-16003: standby database is restricted to read-only access
ORA-16403: shutdown in progress - remote connection is not permitted
ORA-16724: the intended state for resource has been set to OFFLINE
ORA-16903: Unable to connect to database
ORA-16914: Missing connect string. Try \"help\"
ORA-18014: deadlock detected while waiting for resource %s
ORA-21521: exceeded maximum number of connections in OCI (object mode only)
ORA-21522: attempted to use an invalid connection in OCI (object mode only)
ORA-23317: a communication failure has occurred
ORA-24401: cannot open further connections
ORA-24418: Cannot open further sessions.
ORA-24778: cannot open connections
ORA-25400: must replay fetch
ORA-25401: can not continue fetches
ORA-25402: transaction must roll back
ORA-25403: could not reconnect
ORA-25405: transaction status unknown
ORA-25407: connection terminated
ORA-25408: can not safely replay call
ORA-25409: failover happened during the network operation,cannot continue
ORA-25425: connection lost during rollback
ORA-29306: datafile %s is not online
ORA-30006: resource busy; acquire with WAIT timeout expired
ORA-30036: unable to extend segment by %s in undo tablespace '%s'
ORA-30040: Undo tablespace is offline
ORA-31443: deadlock detected while acquiring lock on %s
ORA-37013: (XSACQUIRE_DEADLOCK) Cannot wait to acquire object %j since doing so would cause a deadlock.
ORA-44317: database open read-only