5 Administering Oracle Clusterware Components

About Backing Up and Recovering Voting Disks

The voting disk records node membership information. A node must be able to access more than half of the voting disks at any time. To avoid simultaneous loss of multiple voting disks, each voting disk should be on a storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other voting disks.

For example, if you have five voting disks configured, then a node must be able to access at least three of the voting disks at any time. If a node cannot access the minimum required number of voting disks it is evicted, or removed, from the cluster. After the cause of the failure has been corrected and access to the voting disks has been restored, you can instruct Oracle Clusterware to recover the failed node and restore it to the cluster.

If you lose all copies of the voting disk and do not have a backup, the only safe way to re-create the voting disk is to reinstall Oracle Clusterware. Having a backup of the voting disk can drastically simplify the recovery of your system.

Backing Up Voting Disks

The voting disk files are backed up automatically by Oracle Clusterware if the contents of the files have changed in the following ways:

• Configuration parameters, for example misscount, have been added or modified

• After performing voting disk add or delete operations

Recovering Voting Disks

If a voting disk is damaged, and no longer usable by Oracle Clusterware, you can recreate the voting disk. The voting disk contents are restored from a backup when a new voting file is added; this occurs regardless of whether or not the voting disk file is stored in Oracle Automatic Storage Management (Oracle ASM). If you need to replace a corrupt, damaged, or missing voting disk, then use CRSCTL to first delete the voting disk and then create a new voting disk in the same location.

Restoring a voting disk from a copy created with the operating system dd command is not supported.

Viewing Available OCR Backups

Use the ocrconfig utility to view the backups generated automatically by Oracle Clusterware.

To find the most recent backup of the OCR:

Run the following command on any node in the cluster:

ocrconfig -showbackup

Manually Backing Up the OCR

Use the ocrconfig utility to force Oracle Clusterware to perform a backup of OCR at any time, rather than wait for the automatic backup that occurs at 4-hour intervals. This option is especially useful when you want to obtain a binary backup on demand, such as before you make changes to OCR.

To manually backup the contents of the OCR:

1. Log in as the root user.

2. Use the following command to force Oracle Clusterware to perform an immediate backup of the OCR:

   ocrconfig -manualbackup
   

   The date and identifier of the recently generated OCR backup is displayed.

3. (Optional) If you need to change the location for the OCR backup files, use the following command, where directory_name is the new location for the backups:

   ocrconfig -backuploc directory_name
   

The default location for generating backups on Oracle Enterprise Linux systems is Grid_home/cdata/cluster_name where cluster_name is the name of your cluster and Grid_home is the home directory of your Oracle grid infrastructure software. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the ocrconfig command as part of your operating system backup using standard operating system or third-party tools.

Recovering the OCR

There are two methods for recovering the OCR. The first method uses automatically generated OCR file copies and the second method uses manually created OCR export files.

This section contains the following topics:

• Checking the Status of the OCR

• Restoring the OCR from Automatically Generated OCR Backups

Maintaining the Oracle Local Registry

You maintain the Oracle Local Registry using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities with the -local option.

To check the status of the OLR:

As the root user, use the OCRCHECK utility, as shown in the following example:

[root]# ocrcheck -local

This command produces output similar to the following:

Status of Oracle Local Registry is as follows :
        Version                  :          3
        Total space (kbytes)     :     262132
        Used space (kbytes)      :       9200
        Available space (kbytes) :     252932
        ID                       :  604793089
        Device/File Name         : /u01/grid/cdata/node01.olr
                                    Device/File integrity check succeeded

        Local registry integrity check succeeded

        Logical corruption check succeeded

To view the contents of the OLR:

Use the OCRDUMP utility to display the contents of the OLR to the terminal window that initiated the program, as follows:

ocrdump -local -stdout

To export the OLR to a file:

As the root user, use the OCRCONFIG utility, as shown in the following example:

[root]# ocrconfig –local –export file_name

To import a specified file to the OLR:

As the root user, use the OCRCONFIG utility, as shown in the following example:

[root]# ocrconfig –local –import file_name

To change the location of the OLR file on the local node:

As the root user, use the OCRCONFIG utility to modify the location where the OLR file is stored on the local host:

$ ocrconfig –local –repair -replace current_olr_file_name
-replacement new_olr_file_name

Adding an OCR Location

Oracle Clusterware supports up to 5 OCR copies. You can add an OCR location after an upgrade or after completing the Oracle RAC installation. Additional OCR copies provide greater fault tolerance.

To add an OCR file:

As the root user, enter the following command to add a new OCR file:

[root]# ocrconfig -add new_ocr_file_name 

This command updates the OCR configuration on all the nodes on which Oracle Clusterware is running.

Replacing an OCR

If you need to change the location of an existing OCR, or change the location of a failed OCR to the location of a working one, you can use the following procedure as long as one OCR file remains online.

To change the location of an OCR or replace an OCR file:

1. Use the OCRCHECK utility to verify that a copy of the OCR other than the one you are going to replace is online, using the following command:

   ocrcheck 
   

   Note:

   The OCR that you are replacing can be either online or offline.

2. Use the following command to verify that Oracle Clusterware is running on the node on which the you are going to perform the replace operation:

   crsctl check cluster -all
   

3. As the root user, enter the following command to designate a new location for the specified OCR file:

   [root]# ocrconfig -replace source_ocr_file -replacement destination_ocr_file
   

   This command updates the OCR configuration on all the nodes on which Oracle Clusterware is running.

4. Use the OCRCHECK utility to verify that OCR replacement file is online:

   ocrcheck 
   

Removing an OCR

To remove an OCR file, at least one copy of the OCR must be online. You can remove an OCR location to reduce OCR-related overhead or to stop mirroring your OCR because you moved the OCR to a redundant storage system, such as a redundant array of independent disks (RAID).

To remove an OCR location from your Oracle RAC cluster:

1. Use the OCRCHECK utility to ensure that at least one OCR other than the OCR that you are removing is online.

   ocrcheck
   

   Note:

   Do not perform this OCR removal procedure unless there is at least one active OCR online.

2. As the root user, run the following command on any node in the cluster to remove a specific OCR file:

   [root]# ocrconfig -delete ocr_file_name
   

   This command updates the OCR configuration on all the nodes on which Oracle Clusterware is running.

Repairing an OCR Configuration on a Local Node

If one of the nodes in your cluster was not available when you modified the OCR configuration, then you might need to repair the OCR configuration on that node before it is restarted.

To repair an OCR configuration:

1. As the root user, run the one or more of the following commands on the node on which Oracle Clusterware is stopped, depending on the number and type of changes that were made to the OCR configuration:

   [root]# ocrconfig –repair -add new_ocr_file_name
   

   [root]# ocrconfig –repair -delete ocr_file_name
   

   [root]# ocrconfig –repair -replace source_ocr_file -replacement dest_ocr_file
   

   These commands update the OCR configuration only on the node from which you run the command.

   Note:

   You cannot perform these operations on a node on which the Oracle Clusterware daemon is running.

2. Restart Oracle Clusterware on the node you have just repaired.

3. As the root user, check the OCR configuration integrity of your cluster using the following command:

   [root]# ocrcheck
   

About the OCRCHECK Utility

The OCRCHECK utility displays the data block format version used by the OCR, the available space and used space in the OCR, the ID used for the OCR, and the locations you have configured for the OCR. The OCRCHECK utility calculates a checksum for all the data blocks in all the OCRs that you have configured to verify the integrity of each block. It also returns an individual status for each OCR file as well as a result for the overall OCR integrity check. The following is a sample of the OCRCHECK output:

Status of Oracle Cluster Registry is as follows :
   Version                  :          3
   Total space (kbytes)     :     262144
   Used space (kbytes)      :      16256
   Available space (kbytes) :     245888
   ID                       :  570929253
   Device/File Name         : +CRS_DATA
                              Device/File integrity check succeeded
...
                              Decive/File not configured

   Cluster registry integrity check succeeded

   Logical corruption check succeeded

The OCRCHECK utility creates a log file in the following directory, where Grid_home is the location of the Oracle grid infrastructure installation, and hostname is the name of the local node:

Grid_home/log/hostname/client

The log files have names of the form ocrcheck_nnnnn.log, where nnnnn is the process ID of the operating session that issued the ocrcheck command.

Common Oracle Cluster Registry Problems and Solutions

Table 5-1 describes common OCR problems and their corresponding solutions.