About Oracle Restart

Oracle Restart improves the availability of your Oracle database. When you install Oracle Restart, various Oracle components can be automatically restarted after a hardware or software failure or whenever your database host computer restarts. Table 4-1 lists these components.

Table 4-1 Oracle Components Automatically Restarted by Oracle Restart

Component	Notes
Database instance	Oracle Restart can accommodate multiple databases on a single host computer.
Oracle Net listener	-
Database services	Does not include the default service created upon installation because it is automatically managed by Oracle Database, and does not include any default services created during database creation.
Oracle Automatic Storage Management (Oracle ASM) instance	-
Oracle ASM disk groups	Restarting a disk group means mounting it.
Oracle Notification Services (ONS/eONS)	In a standalone server environment, ONS/eONS can be used in Oracle Data Guard installations for automating failover of connections between primary and standby database through Fast Application Notification (FAN). ONS/eONS is a service for sending FAN events to integrated clients upon failover.

Oracle Restart runs periodic check operations to monitor the health of these components. If a check operation fails for a component, the component is shut down and restarted.

Oracle Restart is used in standalone server (non-clustered) environments only. For Oracle Real Application Clusters (Oracle RAC) environments, the functionality to automatically restart components is provided by Oracle Clusterware.

Oracle Restart runs out of the Oracle Grid Infrastructure home, which you install separately from Oracle Database homes. See the Oracle Database Installation Guide for your platform for information about installing the Oracle Grid Infrastructure home.

Startup Dependencies

Oracle Restart ensures that Oracle components are started in the proper order, in accordance with component dependencies. For example, if database files are stored in Oracle ASM disk groups, then before starting the database instance, Oracle Restart ensures that the Oracle ASM instance is started and the required disk groups are mounted. Likewise, if a component must be shut down, Oracle Restart ensures that dependent components are cleanly shut down first.

Oracle Restart also manages the weak dependency between database instances and the Oracle Net listener (the listener): When a database instance is started, Oracle Restart attempts to start the listener. If the listener startup fails, then the database is still started. If the listener later fails, Oracle Restart does not shut down and restart any database instances.

Manually Starting and Stopping Components

Oracle Restart includes the Server Control (SRVCTL) utility that you use to start and stop Oracle Restart–managed components. When Oracle Restart is in use, Oracle strongly recommends that you use SRVCTL to manually start and stop components.

After you stop a component with SRVCTL, Oracle Restart does not automatically restart that component if a failure occurs. If you then start the component with SRVCTL, that component is again available for automatic restart.

Oracle utilities such as SQL*Plus, the Listener Control utility (LSNRCTL), and ASMCMD are integrated with Oracle Restart. If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database. Similarly, if you shut down the Oracle ASM instance with SQL*Plus or ASMCMD, Oracle Restart does not attempt to restart it.

An important difference between starting a component with SRVCTL and starting it with SQL*Plus (or another utility) is the following:

When you start a component with SRVCTL, any components on which this component depends are automatically started first, and in the proper order.
When you start a component with SQL*Plus (or another utility), other components in the dependency chain are not automatically started; you must ensure that any components on which this component depends are started.

In addition, Oracle Restart enables you to start and stop all of the components managed by Oracle Restart in a specified Oracle home using a single command. The Oracle home can be an Oracle Database home or an Oracle Grid Infrastructure home. This capability is useful when you are installing a patch.

See "Starting and Stopping Components Managed by Oracle Restart" for other differences between starting a component with SRVCTL and starting it with SQL*Plus (or another utility).

Manually Starting and Stopping Oracle Restart

The CRSCTL utility starts and stops Oracle Restart. You can also use the CRSCTL utility to enable or disable Oracle high availability services. Oracle Restart uses Oracle high availability services to start and stop automatically the components managed by Oracle Restart. For example, Oracle high availability services daemons automatically start databases, listeners, and Oracle ASM instances. When Oracle high availability services are disabled, none of the components managed by Oracle Restart are started when a node is rebooted.

Typically, you use the CRSCTL utility when you need to stop all of the running Oracle software in an Oracle installation. For example, you might need to stop Oracle Restart when you are installing a patch or performing operating system maintenance. When the maintenance is complete, you use the CRSCTL utility to start Oracle Restart.

See "Stopping and Restarting Oracle Restart for Maintenance Operations" for information about using the CRSCTL utility.

Oracle Restart Configuration

Oracle Restart maintains a list of all the Oracle components that it manages, and maintains configuration information for each component. All of this information is collectively known as the Oracle Restart configuration. When Oracle Restart starts a component, it starts the component according to the configuration information for that component. For example, the Oracle Restart configuration includes the location of the server parameter file (SPFILE) for databases, and the TCP port to listen on for listeners.

If you install Oracle Restart and then create your database with Database Configuration Assistant (DBCA), DBCA automatically adds the database to the Oracle Restart configuration. When DBCA then starts the database, the required dependencies between the database and other components (for example disk groups in which the database stores data) are established, and Oracle Restart begins to manage the database.

You can manually add and remove components from the Oracle Restart configuration with SRVCTL commands. For example, if you install Oracle Restart onto a host on which a database is already running, you can use SRVCTL to add that database to the Oracle Restart configuration. When you manually add a component to the Oracle Restart configuration and then start it with SRVCTL, Oracle Restart begins to manage the component, restarting it when required.

Note:

Adding a component to the Oracle Restart configuration is also referred to as "registering a component with Oracle Restart."

Other SRVCTL commands enable you to view the status and configuration of Oracle Restart–managed components, temporarily disable and then reenable management for components, and more.

When Oracle Restart is installed, many operations that create Oracle components automatically add the components to the Oracle Restart configuration. Table 4-2 lists some create operations and whether or not the created component is automatically added.

Table 4-2 Create Operations and the Oracle Restart Configuration

Create Operation	Created Component Automatically Added to Oracle Restart Configuration?
Create a database with OUI or DBCA	Yes
Create a database with the `CREATE` `DATABASE` SQL statement	No
Create an Oracle ASM instance with OUI, DBCA, or ASMCA	Yes
Create a disk group (any method)	Yes
Add a listener with NETCA	Yes
Create a database service with SRVCTL	Yes
Create a database service by modifying the `SERVICE_NAMES` initialization parameter^Foot 1	No
Create a database service with `DBMS_SERVICE.CREATE_SERVICE`	No
Create a standby database	No

^Footnote 1Not recommended when Oracle Restart is in use

Table 4-3 lists some delete/drop/remove operations and whether or not the deleted component is also automatically removed from the Oracle Restart configuration.

Table 4-3 Delete/Drop/Remove Operations and the Oracle Restart Configuration

Operation	Deleted Component Automatically Removed from Oracle Restart Configuration?
Delete a database with DBCA	Yes
Delete a database by removing database files with operating system commands^Foot 1	No
Delete a listener with NETCA	Yes
Drop an Oracle ASM disk group (any method)	Yes
Delete a database service with SRVCTL	Yes
Delete a database service by any other means	No

^Footnote 1Not recommended

Oracle Restart Integration with Oracle Data Guard

Oracle Restart is integrated with Oracle Data Guard (Data Guard) and the Oracle Data Guard Broker (the broker). When a database shutdown and restart is required in response to a role change request, Oracle Restart shuts down and restarts the database in an orderly fashion (taking dependencies into account), and according to the settings in the Oracle Restart configuration. Oracle Restart also ensures that, following a Data Guard role transition, all database services configured to run in the new database role are active and all services not configured to run in the new role are stopped.

In addition, the Oracle Restart configuration supports Data Guard–related configuration options for the following components:

Databases—When you add a database to the Oracle Restart configuration, you can specify the current Data Guard role for the database: PRIMARY, PHYSICAL_STANDBY, LOGICAL_STANDBY, or SNAPSHOT_STANDBY. If the role is later changed using the broker, Oracle Restart automatically updates the database configuration with the new role. If you change the database role without using the broker, you must manually modify the database's role in the Oracle Restart configuration to reflect the new role.
Database Services—When adding a database service to the Oracle Restart configuration, you can specify one or more Data Guard roles for the service. When this configuration option is present, upon database restart Oracle Restart starts the service only if one of the service roles matches the current database role.

See Also:

"Configuring Oracle Restart"
"Fast Application Notification with Oracle Restart"
"Automating the Failover of Connections Between Primary and Standby Databases"
Oracle Database Storage Administrator's Guide for information about Oracle Automatic Storage Management
Oracle Real Application Clusters Administration and Deployment Guide for information about automatically restarting databases residing on an Oracle RAC node
Oracle Data Guard Concepts and Administration for information about Oracle Data Guard
Oracle Data Guard Broker for information about FAN events in an Oracle Data Guard environment

Fast Application Notification with Oracle Restart

In a standalone server environment, Oracle Restart uses Oracle Notification Services (ONS) and Oracle Advanced Queues to publish Fast Application Notification (FAN) high availability events. Integrated Oracle clients use FAN to provide fast notification to clients when the service or instance goes down. The client can automate the failover of database connections between a primary database and a standby database.

This section describes how ONS and FAN work with Oracle Restart. It contains the following topics:

Overview of Fast Application Notification
Application High Availability with Services and FAN
Managing Unplanned Outages
Managing Planned Outages
Fast Application Notification High Availability Events
Using Fast Application Notification Callouts
Oracle Clients That Are Integrated with Fast Application Notification

Overview of Fast Application Notification

FAN is a notification mechanism that Oracle Restart can use to notify other processes about configuration changes that include service status changes, such as UP or DOWN events. FAN provides the ability to immediately terminate inflight transaction when an instance or server fails. Integrated Oracle clients receive the events and respond. Applications can respond either by propagating the error to the user or by resubmitting the transactions and masking the error from the application user. When a DOWN event occurs, integrated clients immediately clean up connections to the terminated database. When an UP event occurs, the clients create new connections to the new primary database instance.

Oracle Restart publishes FAN events whenever a managed instance or service goes up or down. After a failover, the Oracle Data Guard Broker (broker) publishes FAN events. These FAN events can be used in the following ways:

Applications can use FAN with Oracle Restart without programmatic changes if they use one of these Oracle integrated database clients: Oracle Database JDBC, Universal Connection Pool for Java, Oracle Call Interface, and Oracle Database ODP.NET. These clients can be configured for Fast Connection Failover (FCF) to automatically connect to a new primary database after a failover.
FAN server-side callouts can be configured on the database tier.

For DOWN events, such as a failed primary database, FAN provides immediate notification to the clients so that they can failover as fast as possible to the new primary database. The clients do not wait for a timeout. The clients are notified immediately, and they must be configured to failover when they are notified.

For UP events, when services and instances are started, new connections can be created so that the application can immediately take advantage of the extra resources.

Through server-side callouts, you can also use FAN to:

Log status information
Page DBAs or open support tickets when resources fail to start
Automatically start dependent external applications that must be co-located with a service

FAN events are published using ONS and Oracle Streams Advanced Queuing queues. The queues are configured automatically when you configure a service. You must configure ONS manually using SRVCTL commands.

The Connection Manager (CMAN) and Oracle Net Services listeners are integrated with FAN events, enabling the CMAN and the listener to immediately de-register services provided by the failed instance and to avoid erroneously sending connection requests to a failed database.

See Also:

The Maximum Availability Architecture (MAA) white paper about client failover:

http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

Application High Availability with Services and FAN

Oracle Database focuses on maintaining service availability. With Oracle Restart, Oracle services are designed to be continuously available. Oracle Restart monitors the database and its services and, when configured, sends event notifications using FAN.

Managing Unplanned Outages

If Oracle Restart detects an outage, then it isolates the failed component and recovers the dependent components. If the failed component is the database instance, then after Oracle Data Guard fails over to the standby database, Oracle Restart on the new primary database starts any services defined with the current role.

FAN events are published by Oracle Restart and the Oracle Data Guard Broker through ONS and Advanced Queuing. You can also perform notifications using FAN callouts.

Note:

Oracle Restart does not run callouts with guaranteed ordering. Callouts are run asynchronously, and they are subject to scheduling variability.

With Oracle Restart, restart and recovery are automatic, including the restarting of the subsystems, such as the listener and the Oracle Automatic Storage Management (Oracle ASM) processes, not just the database. You can use FAN callouts to report faults to your fault management system and to initiate repair jobs.

Managing Planned Outages

For repairs, upgrades, and changes that require you to shut down the primary database, Oracle Restart provides interfaces that disable and enable services to minimize service disruption to application users. Using Oracle Data Guard Broker with Oracle Restart allows a coordinated failover of the database service from the primary to the standby for the duration of the planned outage. Once you complete the operation, you can return the service to normal operation.

The management policy for a service controls whether the service starts automatically when the database is restarted. If the management policy for a service is set to AUTOMATIC, then it restarts automatically. If the management policy for a service is set to MANUAL, then it must be started manually.

Fast Application Notification High Availability Events

Table 4-4 describes the FAN event record parameters and the event types, followed by name-value pairs for the event properties. The event type is always the first entry and the timestamp is always the last entry. In the following example, the name in the name-value pair is shown in Fan event type (service_member), and the value in the name-value pair is shown in Properties:

FAN event type: service_member
Properties: version=1.0 service=ERP database=FINPROD instance=FINPROD host=node1 status=up

Table 4-4 Event Record Parameters and Descriptions

Parameter	Description
`VERSION`	Version of the event record. Used to identify release changes.
`EVENT` `TYPE`	`SERVICE`, `SERVICE_MEMBER`, `DATABASE`, `INSTANCE`, `NODE`, `ASM`, `SRV_PRECONNECT`. Note that database and Instance types provide the database service, such as `DB_UNIQUE_NAME.DB_DOMAIN`.
`DATABASE` `UNIQUE` `NAME`	The unique database supporting the service; matches the initialization parameter value for `DB_UNIQUE_NAME`, which defaults to the value of the initialization parameter `DB_NAME`.
`INSTANCE`	The name of the instance that supports the service; matches the `ORACLE_SID` value.
`NODE` `NAME`	The name of the node that supports the service or the node that has stopped; matches the node name known to Cluster Synchronization Services (CSS).
`SERVICE`	The service name; matches the service in `DBA_SERVICES`.
`STATUS`	Values are `UP`, `DOWN`, `NOT_RESTARTING`, `PRECONN_UP`, `PRECONN_DOWN`, and `UNKNOWN`.
`REASON`	`Data_Guard_Failover`, `Failure`, `Dependency`, `User`, `Autostart`, `Restart`.
`CARDINALITY`	The number of service members that are currently active; included in all `UP` events.
`TIMESTAMP`	The local time zone to use when ordering notification events.

A FAN record matches the database signature of each session as shown in Table 4-5.

Table 4-5 FAN Parameters and Matching Database Signatures

FAN Parameter	Matching Oracle Database Signature
`SERVICE`	`sys_context('userenv', 'service_name')`
`DATABASE` `UNIQUE` `NAME`	`sys_context('userenv', 'db_unique_name')`
`INSTANCE`	`sys_context('userenv', 'instance_name')`
`NODE` `NAME`	`sys_context('userenv', 'server_host')`

Using Fast Application Notification Callouts

FAN callouts are server-side executables that Oracle Restart executes immediately when high availability events occur. You can use FAN callouts to automate the following activities when events occur, such as:

Opening fault tracking tickets
Sending messages to pagers
Sending e-mail
Starting and stopping server-side applications
Maintaining an uptime log by logging each event as it occurs

To use FAN callouts, place an executable in the directory grid_home/racg/usrco on both the primary and the standby database servers. If you are using scripts, then set the shell as the first line of the executable. The following is an example file for the grid_home/racg/usrco/callout.sh callout:

#! /bin/ksh
FAN_LOGFILE= [your path name]/admin/log/`hostname`_uptime.log
echo $* "reported="`date` >> $FAN_LOGFILE &

The following output is from the previous example:

NODE VERSION=1.0 host=sun880-2 status=nodedown reason=
timestamp=08-Oct-2004 04:02:14 reported=Fri Oct 8 04:02:14 PDT 2004

A FAN record matches the database signature of each session, as shown in Table 4-5. Use this information to take actions on sessions that match the FAN event data.

See Also:

Table 4-4 for information about the callout and event details

Oracle Clients That Are Integrated with Fast Application Notification

Oracle has integrated FAN with many of the common Oracle client drivers that are used to connect to Oracle Restart databases. Therefore, the easiest way to use FAN is to use an integrated Oracle Client.

You can use the CMAN session pools, Oracle Call Interface, Universal Connection Pool for Java, JDBC simplefan API, and ODP.NET connection pools. The overall goal is to enable applications to consistently obtain connections to the available primary database at anytime.