Skip Headers
Oracle® Data Guard Concepts and Administration
11g Release 2 (11.2)

Part Number E10700-01
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

6 Redo Transport Services

This chapter describes how to configure and monitor Oracle redo transport services. The following topics are discussed:

6.1 Introduction to Redo Transport Services

Redo transport services performs the automated transfer of redo data between Oracle databases. The following redo transport destinations are supported:

An Oracle database can send redo data to up to thirty redo transport destinations. Each redo transport destination is individually configured to receive redo data via one of two redo transport modes:

6.2 Configuring Redo Transport Services

This section describes how to configure redo transport services. The following topics are discussed:

The section is written at a level of detail that assumes that the reader has a thorough understanding of the following topics, which are described in the Oracle Database Administrator's Guide, Oracle Database Backup and Recovery User's Guide, and Oracle Database Net Services Administrator's Guide:

6.2.1 Redo Transport Security

Redo transport uses Oracle Net sessions to transport redo data. These redo transport sessions are authenticated using either the Secure Socket Layer (SSL) protocol or a remote login password file.

6.2.1.1 Redo Transport Authentication Using SSL

Secure Sockets Layer (SSL) is an industry standard protocol for securing network connections. SSL uses RSA public key cryptography and symmetric key cryptography to provide authentication, encryption, and data integrity. SSL is automatically used for redo transport authentication between two Oracle databases if:

  • The databases are members of the same Oracle Internet Directory (OID) enterprise domain and that domain allows the use of current user database links.

  • The LOG_ARCHIVE_DEST_n, and FAL_SERVER database initialization parameters that correspond to the databases use Oracle Net connect descriptors configured for SSL.

  • Each database has an Oracle wallet or a supported hardware security module that contains a user certificate with a distinguished name (DN) that matches the DN in the OID entry for the database.

See Also:

6.2.1.2 Redo Transport Authentication Using a Password File

If the SSL authentication requirements are not met, each database must use a remote login password file. In a Data Guard configuration, all physical and snapshot standby databases must use a copy of the password file from the primary database, and that copy must be refreshed whenever the SYSOPER or SYSDBA privilege is granted or revoked, and after the password of any user with these privileges is changed.

When a password file is used for redo transport authentication, the password of the user account used for redo transport authentication is compared between the database initiating a redo transport session and the target database. The password must be the same at both databases to create a redo transport session.

By default, the password of the SYS user is used to authenticate redo transport sessions when a password file is used. The REDO_TRANSPORT_USER database initialization parameter can be used to select a different user password for redo transport authentication by setting this parameter to the name of any user who has been granted the SYSOPER privilege. For administrative ease, Oracle recommends that the REDO_TRANSPORT_USER parameter be set to the same value on the redo source database and at each redo transport destination.

See Also:

Oracle Database Administrator's Guide for more information creating and maintaining remote login password files

6.2.2 Configuring an Oracle Database to Send Redo Data

This section describes how to configure an Oracle database to send redo data to a redo transport destination.

The LOG_ARCHIVE_DEST_n database initialization parameter (where n is an integer from 1 to 31) is used to specify the location of a local archive redo log or to specify a redo transport destination. This section describes the latter use of this parameter.

There is a LOG_ARCHIVE_DEST_STATE_n database initialization parameter (where n is an integer from 1 to 31) that corresponds to each LOG_ARCHIVE_DEST_n parameter. This parameter is used to enable or disable the corresponding redo destination. Table 6-1 shows the valid values that can be assigned to this parameter.

Table 6-1 LOG_ARCHIVE_DEST_STATE_n Initialization Parameter Values

Value Description

ENABLE

Redo transport services can transmit redo data to this destination. This is the default.

DEFER

Redo transport services will not transmit redo data to this destination.

ALTERNATE

This destination will become enabled if communication to its associated destination fails.


A redo transport destination is configured by setting the LOG_ARCHIVE_DEST_n parameter to a character string that includes one or more attributes. This section briefly describes the most commonly used attributes. See Chapter 15 for a full description of all LOG_ARCHIVE_DEST_n parameter attributes.

The SERVICE attribute, which is a mandatory attribute for a redo transport destination, must be the first attribute specified in the attribute list. The SERVICE attribute is used to specify the Oracle Net service name used to connect to the redo transport destination. The service name must be resolvable through an Oracle Net naming method to an Oracle Net connect descriptor that matches the Oracle Net listener(s) at the redo transport destination. The connect descriptor must specify that a dedicated server connection be used, unless that is the default connection type for the redo transport destination.

See Also:

Oracle Database Net Services Administrator's Guide for information about Oracle Net service names, connect descriptors, listeners, and network security

The SYNC attribute is used to specify that the synchronous redo transport mode be used to send redo data to a redo transport destination.

The ASYNC attribute is used to specify that the asynchronous redo transport mode be used to send redo data to a redo transport destination. The asynchronous redo transport mode will be used if neither the SYNC nor the ASYNC attribute is specified.

The NET_TIMEOUT attribute is used to specify how long the LGWR process will block waiting for an acknowledgement that redo data has been successfully received by a destination that uses the synchronous redo transport mode. If an acknowledgement is not received within NET_TIMEOUT seconds, the redo transport connection is terminated and an error is logged.

Oracle recommends that the NET_TIMEOUT attribute be specified whenever the synchronous redo transport mode is used, so that the maximum duration of a redo source database stall caused by a redo transport fault can be precisely controlled. See Section 6.3.2 for information about monitoring synchronous redo transport mode response time.

The AFFIRM attribute is used to specify that redo received from a redo source database is not acknowledged until it has been written to the standby redo log. The NOAFFIRM attribute is used to specify that received redo is acknowledged without waiting for received redo to be written to the standby redo log.

The DB_UNIQUE_NAME attribute is used to specify the DB_UNIQUE_NAME of a redo transport destination. The DB_UNIQUE_NAME attribute must be specified if the LOG_ARCHIVE_CONFIG database initialization parameter has been defined and its value includes a DG_CONFIG list.

If the DB_UNIQUE_NAME attribute is specified, its value must match one of the DB_UNIQUE_NAME values in the DG_CONFIG list. It must also match the value of the DB_UNIQUE_NAME database initialization parameter at the redo transport destination. If either match fails, an error is logged and redo transport will not be possible to that destination.

The VALID_FOR attribute is used to specify when redo transport services transmits redo data to a redo transport destination. Oracle recommends that the VALID_FOR attribute be specified for each redo transport destination at every site in a Data Guard configuration so that redo transport services will continue to send redo data to all standby databases after a role transition, regardless of which standby database assumes the primary role.

The REOPEN attribute is used to specify the minimum number of seconds between automatic reconnect attempts to a redo transport destination that is inactive because of a previous error.

The COMPRESSION attribute is used to specify that redo data is transmitted to a redo transport destination in compressed form. Redo transport compression can significantly improve redo transport performance on network links with low bandwidth and high latency.

Redo transport compression is a feature of the Oracle Advanced Compression option. You must purchase a license for this option before using the redo transport compression feature.

The following example uses all of the LOG_ARCHIVE_DEST_n attributes described in this section. A DB_UNIQUE_NAME has been specified for both destinations, as has the use of compression. If a redo transport fault occurs at either destination, redo transport will attempt to reconnect to that destination, but not more frequently than once every 60 seconds.

DB_UNIQUE_NAME=BOSTON
LOG_ARCHIVE_CONFIG='DG_CONFIG=(BOSTON,CHICAGO,HARTFORD)' 
LOG_ARCHIVE_DEST_2='SERVICE=CHICAGO ASYNC NOAFFIRM VALID_FOR=(ONLINE_LOGFILE, 
PRIMARY_ROLE) REOPEN=60 COMPRESSION=ENABLE  DB_UNIQUE_NAME=CHICAGO' 
LOG_ARCHIVE_DEST_STATE_2='ENABLE' 
LOG_ARCHIVE_DEST_3='SERVICE=HARTFORD SYNC AFFIRM NET_TIMEOUT=30 
VALID_FOR=(ONLINE_LOGFILE,PRIMARY_ROLE) REOPEN=60 COMPRESSION=ENABLE   
DB_UNIQUE_NAME=HARTFORD' 
LOG_ARCHIVE_DEST_STATE_3='ENABLE'

6.2.2.1 Viewing Attributes With V$ARCHIVE_DEST

The V$ARCHIVE_DEST view can be queried to see the current settings and status for each redo transport destination.

6.2.3 Configuring an Oracle Database to Receive Redo Data

This section describes how to configure a redo transport destination to receive and to archive redo data from a redo source database.

The following topics are discussed:

6.2.3.1 Creating and Managing a Standby Redo Log

The synchronous and asynchronous redo transport modes require that a redo transport destination have a standby redo log. A standby redo log is used to store redo received from another Oracle database. Standby redo logs are structurally identical to redo logs, and are created and managed using the same SQL statements used to create and manage redo logs.

Redo received from another Oracle database via redo transport is written to the current standby redo log group by a RFS background process. When a log switch occurs on the redo source database, incoming redo is then written to the next standby redo log group, and the previously used standby redo log group is archived by an ARCn background process.

The process of sequentially filling and then archiving redo log file groups at a redo source database is mirrored at each redo transport destination by the sequential filling and archiving of standby redo log groups.

Each standby redo log file must be at least as large as the largest redo log file in the redo log of the redo source database. For administrative ease, Oracle recommends that all redo log files in the redo log at the redo source database and the standby redo log at a redo transport destination be of the same size.

The standby redo log must have at least one more redo log group than the redo log on the redo source database.

Perform the following query on a redo source database to determine the size of each log file and the number of log groups in the redo log:

SQL> SELECT GROUP#, BYTES FROM V$LOG;

Perform the following query on a redo destination database to determine the size of each log file and the number of log groups in the standby redo log:

SQL> SELECT GROUP#, BYTES FROM V$STANDBY_LOG;

Oracle recommends that a standby redo log be created on the primary database in a Data Guard configuration so that it is immediately ready to receive redo data following a switchover to the standby role.

The ALTER DATABASE ADD STANDBY LOGFILE SQL statement is used to create a standby redo log and to add standby redo log groups to an existing standby redo log.

For example, assume that the redo log on the redo source database has two redo log groups and that each of those contain one 500 MB redo log file. In this case, the standby redo log should have at least 3 standby redo log groups to satisfy the requirement that a standby redo log must have at least one more redo log group than the redo log at the redo source database.

The following SQL statements might be used to create a standby redo log that is appropriate for the previous scenario:

ALTER DATABASE ADD STANDBY LOGFILE
   ('/oracle/dbs/slog1.rdo') SIZE 500M;
 
ALTER DATABASE ADD STANDBY LOGFILE
   ('/oracle/dbs/slog2.rdo') SIZE 500M;
 
ALTER DATABASE ADD STANDBY LOGFILE
   ('/oracle/dbs/slog3.rdo') SIZE 500M;

Caution:

Whenever a redo log group is added to the primary database in an Oracle Data Guard configuration, a standby redo log group must also be added to the standby redo log at each standby database in the configuration that uses the synchronous redo transport mode. If this is not done, a primary database that is running in the maximum protection data protection mode may shut down, and a primary database that is running in the maximum availability data protection mode may shift to the maximum performance data protection mode.

6.2.3.2 Configuring Standby Redo Log Archival

This section describes how to configure standby redo log archival.

See Also:

6.2.3.2.1 Enable Archiving

If archiving is not enabled, issue the following statements to put the database in ARCHIVELOG mode and to enable automatic archiving:

SQL> SHUTDOWN IMMEDIATE;
SQL> STARTUP MOUNT;
SQL> ALTER DATABASE ARCHIVELOG;

Note that the database must be in ARCHIVELOG mode for standby redo log archival to be performed.

6.2.3.2.2 Standby Redo Log Archival to a fast recovery area

Take the following steps to set up standby redo log archival to a fast recovery area:

  1. Set the LOCATION attribute of a LOG_ARCHIVE_DEST_n parameter to USE_DB_RECOVERY_FILE_DEST.

  2. Set the VALID_FOR attribute of the same LOG_ARCHIVE_DEST_n parameter to a value that allows standby redo log archival.

The following are some sample parameter values that might be used to configure a physical standby database to archive its standby redo log to the fast recovery area:

LOG_ARCHIVE_DEST_2 = 'LOCATION=USE_DB_RECOVERY_FILE_DEST
VALID_FOR=(STANDBY_LOGFILE,STANDBY_ROLE)'
LOG_ARCHIVE_DEST_STATE_2=ENABLE

Oracle recommends the use of a fast recovery area, because it simplifies the management of archived redo log files.

6.2.3.2.3 Standby Redo Log Archival to a Local FIle System Location

Take the following steps to set up standby redo log archival to a local file system location:

  1. Set the LOCATION attribute of a LOG_ARCHIVE_DEST_n parameter to a valid pathname.

  2. Set the VALID_FOR attribute of the same LOG_ARCHIVE_DEST_n parameter to a value that allows standby redo log archival.

The following are some sample parameter values that might be used to configure a physical standby database to archive its standby redo log to a local file system location:

LOG_ARCHIVE_DEST_2 = 'LOCATION = /disk2/archive
VALID_FOR=(STANDBY_LOGFILE,STANDBY_ROLE)'
LOG_ARCHIVE_DEST_STATE_2=ENABLE

6.2.3.3 Cases Where Redo Is Written Directly To an Archived Redo Log File

Redo received by a standby database is written directly to an archived redo log file if a standby redo log group is not available or if the redo was sent to resolve a redo gap. When this occurs, redo is written to the location specified by the LOCATION attribute of one LOG_ARCHIVE_DEST_n parameter that is valid for archiving redo received from another database. The LOG_ARCHIVE_DEST_n parameter that is used for this purpose is determined when the standby database is mounted, and this choice is reevaluated each time a LOG_ARCHIVE_DEST_n parameter is modified.

6.3 Monitoring Redo Transport Services

This section discusses the following topics:

6.3.1 Monitoring Redo Transport Status

This section describes the steps used to monitor redo transport status on a redo source database.

Step 1   Determine the most recently archived redo log file.

Perform the following query on the redo source database to determine the most recently archived sequence number for each thread:

SQL> SELECT MAX(SEQUENCE#), THREAD# FROM V$ARCHIVED_LOG GROUP BY THREAD#;
Step 2   Determine the most recently archived redo log file at each redo transport destination.

Perform the following query on the redo source database to determine the most recently archived redo log file at each redo transport destination:

SQL> SELECT DESTINATION, STATUS, ARCHIVED_THREAD#, ARCHIVED_SEQ# -
> FROM V$ARCHIVE_DEST_STATUS -
> WHERE STATUS <> 'DEFERRED' AND STATUS <> 'INACTIVE';
 
DESTINATION         STATUS  ARCHIVED_THREAD#  ARCHIVED_SEQ#
------------------  ------  ----------------  -------------
/private1/prmy/lad   VALID                 1            947
standby1             VALID                 1            947

The most recently archived redo log file should be the same for each destination. If it is not, a status other than VALID may identify an error encountered during the archival operation to that destination.

Step 3   Find out if archived redo log files have been received at a redo transport destination.

A query can be performed at a redo source database to find out if an archived redo log file has been received at a particular redo transport destination. Each destination has an ID number associated with it. You can query the DEST_ID column of the V$ARCHIVE_DEST view on a database to identify each destination's ID number.

Assume that destination 1 points to the local archived redo log and that destination 2 points to a redo transport destination. Perform the following query at the redo source database to find out if any log files are missing at the redo transport destination:

SQL> SELECT LOCAL.THREAD#, LOCAL.SEQUENCE# FROM -
> (SELECT THREAD#, SEQUENCE# FROM V$ARCHIVED_LOG WHERE DEST_ID=1) -
> LOCAL WHERE -
> LOCAL.SEQUENCE# NOT IN -
> (SELECT SEQUENCE# FROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND -
> THREAD# = LOCAL.THREAD#);
 
THREAD#    SEQUENCE#
---------  ---------
  1        12
  1        13
  1        14
Step 4   Trace the progression of redo transmitted to a redo transport destination.

Set the LOG_ARCHIVE_TRACE database initialization parameter at a redo source database and at each redo transport destination to trace redo transport progress. See Appendix G for complete details and examples.

6.3.2 Monitoring Synchronous Redo Transport Response Time

The V$REDO_DEST_RESP_HISTOGRAM view contains response time data for each redo transport destination. This response time data is maintained for redo transport messages sent via the synchronous redo transport mode.

The data for each destination consists of a series of rows, with one row for each response time. To simplify record keeping, response times are rounded up to the nearest whole second for response times less than 300 seconds. Response times greater than 300 seconds are round up to 600, 1200, 2400, 4800, or 9600 seconds.

Each row contains four columns: FREQUENCY, DURATION, DEST_ID, and TIME.

The FREQUENCY column contains the number of times that a given response time has been observed. The DURATION column corresponds to the response time. The DEST_ID column identifies the destination. The TIME column contains a timestamp taken when the row was last updated.

The response time data in this view is useful for identifying synchronous redo transport mode performance issues that can affect transaction throughput on a redo source database. It is also useful for tuning the NET_TIMEOUT attribute.

The next three examples show example queries for destination 2, which corresponds to the LOG_ARCHIVE_DEST_2 parameter. To display response time data for a different destination, simply change the DEST_ID in the query.

Perform the following query on a redo source database to display the response time histogram for destination 2:

SQL> SELECT FREQUENCY, DURATION FROM -
> V$REDO_DEST_RESP_HISTOGRAM WHERE DEST_ID=2 AND FREQUENCY>1;

Perform the following query on a redo source database to display the slowest response time for destination 2:

SQL> SELECT max(DURATION) FROM V$REDO_DEST_RESP_HISTOGRAM -
> WHERE DEST_ID=2 AND FREQUENCY>1;

Perform the following query on a redo source database to display the fastest response time for destination 2:

SQL> SELECT min( DURATION) FROM V$REDO_DEST_RESP_HISTOGRAM -
> WHERE DEST_ID=2 AND FREQUENCY>1;

Note:

The highest observed response time for a destination cannot exceed the highest specified NET_TIMEOUT value specified for that destination, because synchronous redo transport mode sessions are terminated if a redo transport destination does not respond to a redo transport message within NET_TIMEOUT seconds.

6.3.3 Redo Gap Detection and Resolution

A redo gap occurs whenever redo transmission is interrupted. When redo transmission resumes, redo transport services automatically detects the redo gap and resolves it by sending the missing redo to the destination.

The time needed to resolve a redo gap is directly proportional to the size of the gap and inversely proportional to the effective throughput of the network link between the redo source database and the redo transport destination. Redo transport services has two options that may reduce redo gap resolution time when low performance network links are used:

  • Redo Transport Compression

    The COMPRESSION attribute of the LOG_ARCHIVE_DEST_n parameter is used to specify that redo data be compressed before transmission to the destination.

  • Parallel Redo Transport Network Sessions

    The MAX_CONNECTIONS attribute of the LOG_ARCHIVE_DEST_n parameter can be used to specify that more than one network session be used to send the redo needed to resolve a redo gap.

See Chapter 15, "LOG_ARCHIVE_DEST_n Parameter Attributes" for more information about the COMPRESSION and MAX_CONNECTIONS attributes.

6.3.3.1 Manual Gap Resolution

In some situations, gap resolution cannot be performed automatically and it must be performed manually. For example, redo gap resolution must be performed manually on a logical standby database if the primary database is unavailable.

Perform the following query at the physical standby database to determine if there is redo gap on a physical standby database:

SQL> SELECT * FROM V$ARCHIVE_GAP;

    THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
-----------  -------------  --------------
          1              7              10

The output from the previous example indicates that the physical standby database is currently missing log files from sequence 7 to sequence 10 for thread 1.

Perform the following query on the primary database to locate the archived redo log files on the primary database (assuming the local archive destination on the primary database is LOG_ARCHIVE_DEST_1):

SQL> SELECT NAME FROM V$ARCHIVED_LOG WHERE THREAD#=1 AND -
> DEST_ID=1 AND SEQUENCE# BETWEEN 7 AND 10;

NAME
--------------------------------------------------------------------------------
/primary/thread1_dest/arcr_1_7.arc 
/primary/thread1_dest/arcr_1_8.arc 
/primary/thread1_dest/arcr_1_9.arc

Note:

This query may return consecutive sequences for a given thread. In that case, there is no actual gap, but the associated thread was disabled and enabled within the time period of generating these two archived logs. The query also does not identify the gap that may exist at the tail end for a given thread. For instance, if the primary database has generated archived logs up to sequence 100 for thread 1, and the latest archived log that the logical standby database has received for the given thread is the one associated with sequence 77, this query will not return any rows, although we have a gap for the archived logs associated with sequences 78 to 100.

Copy these log files to the physical standby database and register them using the ALTER DATABASE REGISTER LOGFILE. For example:

SQL> ALTER DATABASE REGISTER LOGFILE -
> '/physical_standby1/thread1_dest/arcr_1_7.arc';

SQL> ALTER DATABASE REGISTER LOGFILE -
> '/physical_standby1/thread1_dest/arcr_1_8.arc';

SQL> ALTER DATABASE REGISTER LOGFILE -
> '/physical_standby1/thread1_dest/arcr_1_9.arc';

Note:

The V$ARCHIVE_GAP view on a physical standby database only returns the gap that is currently blocking Redo Apply from continuing. After resolving the gap, query the V$ARCHIVE_GAP view again on the physical standby database to determine if there is another gap sequence. Repeat this process until there are no more gaps.

To determine if there is a redo gap on a logical standby database, query the DBA_LOGSTDBY_LOG view on the logical standby database. For example, the following query indicates there is a gap in the sequence of archived redo log files because it displays two files for THREAD 1 on the logical standby database. (If there are no gaps, the query will show only one file for each thread.) The output shows that the highest registered file is sequence number 10, but there is a gap at the file shown as sequence number 6:

SQL> COLUMN FILE_NAME FORMAT a55
SQL> SELECT THREAD#, SEQUENCE#, FILE_NAME FROM DBA_LOGSTDBY_LOG L -
> WHERE NEXT_CHANGE# NOT IN -
> (SELECT FIRST_CHANGE# FROM DBA_LOGSTDBY_LOG WHERE L.THREAD# = THREAD#) -
> ORDER BY THREAD#, SEQUENCE#;
 
   THREAD#  SEQUENCE# FILE_NAME
---------- ---------- -----------------------------------------------
         1          6 /disk1/oracle/dbs/log-1292880008_6.arc
         1         10 /disk1/oracle/dbs/log-1292880008_10.arc

Copy the missing log files, with sequence numbers 7, 8, and 9, to the logical standby system and register them using the ALTER DATABASE REGISTER LOGICAL LOGFILE statement. For example:

SQL> ALTER DATABASE REGISTER LOGICAL LOGFILE -
> '/disk1/oracle/dbs/log-1292880008_7.arc'; 

SQL> ALTER DATABASE REGISTER LOGICAL LOGFILE -
> '/disk1/oracle/dbs/log-1292880008_8.arc';

SQL> ALTER DATABASE REGISTER LOGICAL LOGFILE -
> '/disk1/oracle/dbs/log-1292880008_9.arc';

Note:

A query based on the DBA_LOGSTDBY_LOG view on a logical standby database, as specified above, only returns the gap that is currently blocking SQL Apply from continuing. After resolving the gap, query the DBA_LOGSTDBY_LOG view again on the logical standby database to determine if there is another gap sequence. Repeat this process until there are no more gaps.

6.3.4 Redo Transport Services Wait Events

Table 6-2 lists several of the Oracle wait events used to track redo transport wait time on a redo source database. These wait events are found in the V$SYSTEM_EVENT dynamic performance view.

For a complete list of the Oracle wait events used by redo transport, see the Oracle Data Guard Redo Transport and Network Best Practices white paper on the Oracle Maximum Availability Architecture (MAA) home page at:

http://otn.oracle.com/deploy/availability/htdocs/maa.htm

Table 6-2 Redo Transport Wait Events

Wait Event Description

LNS wait on ATTACH

Total time spent waiting for redo transport sessions to be established to all ASYNC and SYNC redo transport destinations

LNS wait on SENDREQ

Total time spent waiting for redo data to be written to all ASYNC and SYNC redo transport destinations

LNS wait on DETACH

Total time spent waiting for redo transport connections to be terminated to all ASYNC and SYNC redo transport destinations


6.4 Tuning Redo Transport

The Oracle Data Guard Redo Transport and Network Configuration Best Practices white paper describes how to optimize redo transport for best performance. This paper is available on the Oracle Maximum Availability Architecture (MAA) home page at:

http://otn.oracle.com/deploy/availability/htdocs/maa.htm