1 Introduction to Oracle Clusterware

Oracle Clusterware Hardware Concepts and Requirements

A cluster comprises one or more servers. The hardware in a server in a cluster (or cluster member or node) is similar to a standalone server. However, a server that is part of a cluster, otherwise known as a node or a cluster member, requires a second network. This second network is referred to as the interconnect. For this reason, cluster member nodes require at least two network interface cards: one for a public network and one for a private network. The interconnect network is a private network using a switch (or multiple switches) that only the nodes in the cluster can access.^Foot 1 

Cluster size is determined by the requirements of the workload running on the cluster and the number of nodes that you have configured in the cluster. If you are implementing a cluster for high availability, then configure redundancy for all of the components of the infrastructure as follows:

• At least two network interfaces for the public network, bonded to provide one address

• At least two network interfaces for the private interconnect network, also bonded to provide one address

The cluster requires cluster-aware storage^Foot 2  that is connected to each server in the cluster. This may also be referred to as a multihost device. Oracle Clusterware supports NFS, iSCSI, Direct Attached Storage (DAS), Storage Area Network (SAN) storage, and Network Attached Storage (NAS).

To provide redundancy for storage, generally provide at least two connections from each server to the cluster-aware storage. There may be more connections depending on your I/O requirements. It is important to consider the I/O requirements of the entire cluster when choosing your storage subsystem.

Most servers have at least one local disk that is internal to the server. Often, this disk is used for the operating system binaries; you can also use this disk for the Oracle software binaries. The benefit of each server having its own copy of the Oracle binaries is that it increases high availability, so that corruption to a one binary does not affect all of the nodes in the cluster simultaneously. It also allows rolling upgrades, which reduce downtime.

Oracle Clusterware Operating System Concepts and Requirements

Each server must have an operating system that is certified with the Oracle Clusterware version you are installing. Refer to the certification matrices available on My Oracle Support (formerly OracleMetaLink) for details, which are available from the following URL:

http://certify.oraclecorp.com/certifyv3/certify/cert_views.group_selection?p_html_source=0

When the operating system is installed and working, you can then install Oracle Clusterware to create the cluster. Oracle Clusterware is installed independently of Oracle Database. Once Oracle Clusterware is installed, you can then install Oracle Database or Oracle RAC on any of the nodes in the cluster.

Oracle Clusterware Software Concepts and Requirements

Oracle Clusterware uses voting disk files to provide fencing and cluster node membership determination. The OCR provides cluster configuration information. You can place the Oracle Clusterware files on either Oracle ASM or on shared common disk storage. If you configure Oracle Clusterware on storage that does not provide file redundancy, then Oracle recommends that you configure multiple locations for OCR and voting disks. The voting disks and OCR are described as follows:

• Voting Disks

  Oracle Clusterware uses voting disk files to determine which nodes are members of a cluster. You can configure voting disks on Oracle ASM, or you can configure voting disks on shared storage.

  If you configure voting disks on Oracle ASM, then you do not need to manually configure the voting disks. Depending on the redundancy of your disk group, an appropriate number of voting disks are created.

  If you do not configure voting disks on Oracle ASM, then for high availability, Oracle recommends that you have a minimum of three voting disks on physically separate storage. This avoids having a single point of failure. If you configure a single voting disk, then you must use external mirroring to provide redundancy.

  You should have at least three voting disks, unless you have a storage device, such as a disk array that provides external redundancy. Oracle recommends that you do not use more than five voting disks. The maximum number of voting disks that is supported is 15.

• Oracle Cluster Registry

  Oracle Clusterware uses the Oracle Cluster Registry (OCR) to store and manage information about the components that Oracle Clusterware controls, such as Oracle RAC databases, listeners, virtual IP addresses (VIPs), and services and any applications. The OCR stores configuration information in a series of key-value pairs in a tree structure. To ensure cluster high availability, Oracle recommends that you define multiple OCR locations (multiplex). In addition:

  • You can have up to five OCR locations

  • Each OCR location must reside on shared storage that is accessible by all of the nodes in the cluster

  • You can replace a failed OCR location online if it is not the only OCR location

  • You must update the OCR through supported utilities such as Oracle Enterprise Manager, the Server Control Utility (SRVCTL), the OCR configuration utility (OCRCONFIG), or the Database Configuration Assistant (DBCA)

  See Also:

  Chapter 2, "Administering Oracle Clusterware" for more information about voting disks and the OCR

Oracle Clusterware Network Configuration Concepts

Oracle Clusterware enables a dynamic grid infrastructure through the self-management of the network requirements for the cluster. Oracle Clusterware 11g release 2 (11.2) supports the use of DHCP for all private interconnect addresses, as well as for most of the VIP addresses. DHCP provides dynamic configuration of the host's IP address, but it does not provide an optimal method of producing names that are useful to external clients.

When you are using Oracle RAC, all of the clients must be able to reach the database. This means that the VIP addresses must be resolved by the clients. This problem is solved by the addition of the Grid Naming Service (GNS) to the cluster. GNS is linked to the corporate Domain Name Service (DNS) so that clients can easily connect to the cluster and the databases running there. Activating GNS in a cluster requires a DHCP service on the public network.

# Delegate to gns on mycluster
mycluster.example.com NS myclustergns.example.com
#Let the world know to go to the GNS vip
myclustergns.example.com. 10.9.8.7

Upgrading Oracle Clusterware

Oracle supports in-place and out-of-place upgrades. Both strategies facilitate rolling upgrades. For Oracle Clusterware 11g release 2 (11.2), in-place upgrades are supported for patches only. Patch bundles and one-off patches are supported for in-place upgrades but patch sets and major point releases are supported for out-of-place upgrades only.

An in-place upgrade replaces the Oracle Clusterware software with the newer version in the same Grid home. Out-of-place upgrade has both versions of the same software present on the nodes at the same time, in different Grid homes, but only one version is active.

Rolling upgrades avoid downtime and ensure continuous availability of Oracle Clusterware while the software is upgraded to the new version. When you upgrade to 11g release 2 (11.2), Oracle Clusterware and Oracle ASM binaries are installed as a single binary called the grid infrastructure. You can upgrade Oracle Clusterware in a rolling manner from Oracle Clusterware 10g and Oracle Clusterware 11g, however you can only upgrade Oracle ASM in a rolling manner from Oracle Database 11g release 1 (11.1).

The Oracle Clusterware Stack

Oracle Clusterware consists of two separate stacks: an upper stack anchored by the Cluster Ready Services (CRS) daemon (crsd) and a lower stack anchored by the Oracle High Availability Services daemon (ohasd). These two stacks have several processes that facilitate cluster operations. The following sections describe these stacks in more detail:

• The Cluster Ready Services Stack

• The Oracle High Availability Services Stack

Oracle Clusterware Processes on Linux and UNIX Systems

Oracle Clusterware processes on Linux and UNIX systems include the following:

• crsd: Performs high availability recovery and management operations such as maintaining the OCR and managing application resources. This grid infrastructure process runs as root and restarts automatically upon failure.

  When you install Oracle Clusterware in a single-instance database environment for Oracle ASM and Oracle Restart, ohasd manages application resources and crsd is not used.

• cssdagent: Starts, stops, and checks the status of the CSS daemon, ocssd. In addition, the cssdagent and cssdmonitor provide the following services to guarantee data integrity:

  • Monitors the CSS daemon; if the CSS daemon stops, then it shuts down the node

  • Monitors the node scheduling to verify that the node is not hung, and shuts down the node on recovery from a hang.

  oclskd: (Oracle Clusterware Kill Daemon) CSS uses this daemon to stop processes associated with CSS group members for which stop requests have come in from other members on remote nodes.

• ctssd: Cluster time synchronization service daemon: Synchronizes the time on all of the nodes in a cluster to match the time setting on the master node but not to an external clock.

  When you install Oracle Clusterware, the Cluster Time Synchronization Service (CTSS) is installed as part of the software package. During installation, the Cluster Verification Utility (CVU) determines if the network time protocol (NTP) is in use on any nodes in the cluster. On Windows systems, CVU checks for NTP and Windows Time Service.

  If Oracle Clusterware finds that NTP is running or that NTP has been configured, then NTP is not affected by the CTSS installation. Instead, CTSS starts in observer mode (this condition is logged in the alert log for Oracle Clusterware). CTSS then monitors the cluster time and logs alert messages, if necessary, but CTSS does not modify the system time.

  If Oracle Clusterware detects that NTP is not running and is not configured, then CTSS designates one node as a clock reference, and synchronizes all of the other cluster member time and date settings to those of the clock reference.

  Oracle Clusterware considers an NTP installation to be misconfigured if one of the following is true:

  • NTP is not installed on some nodes of the cluster; CVU detects an NTP installation by a configuration file, such as ntp.conf

  • The primary and alternate clock references are different for all of the nodes of the cluster

  • The NTP processes are not running on all of the nodes of the cluster

  Only one type of time synchronization service can be active on the cluster.

  See Also:

  "Cluster Time Management" for more information about CTSS

• diskmon (Disk Monitor daemon): Monitors and performs I/O fencing for HP Oracle Exadata Storage Server storage. Because Exadata storage can be added to any Oracle RAC node at any time, the diskmon daemon is always started when ocssd starts.

• evmd (Event manager daemon): Distributes and communicates some cluster events to all of the cluster members so that they are aware of changes in the cluster.

  evmlogger (Event manager logger): This is started by EVMD at startup. This reads a configuration file to determine what events to subscribe to from EVMD and it runs user defined actions for those events. This facility maintains backward compatibility only.

• gpnpd (Grid Plug and Play daemon): Manages distribution and maintenance of the Grid Plug and Play profile containing cluster definition data.

• mdnsd (Multicast Domain Name Service daemon): Manages name resolution and service discovery within attached subnets.

• ocssd (Cluster Synchronization Service daemon): Manages cluster node membership and runs as the oracle user; failure of this process results in a node restart.

• ohasd (Oracle High Availability Services daemon): Starts Oracle Clusterware processes and also manages the OLR and acts as the OLR server, as shown in Figure 1-2.

  In a cluster, ohasd runs as root. However, in an Oracle Restart environment, where ohasd manages application resources, it runs as the oracle user.

Figure 1-2 Cluster Startup

Description of "Figure 1-2 Cluster Startup"

Oracle Clusterware Version Compatibility

You can install different releases of Oracle Clusterware, Oracle ASM, and Oracle Database on your cluster. Follow these guidelines when installing different releases of software on your cluster:

• You can only have one installation of Oracle Clusterware running in a cluster, and it must be installed into its own home (Grid_home). The release of Oracle Clusterware that you use must be equal to or higher than the Oracle ASM and Oracle RAC versions that are running in the cluster. You cannot install a version of Oracle RAC that was released after the version of Oracle Clusterware that you run on the cluster. In other words:

  • Oracle Clusterware 11g release 2 (11.2) supports Oracle ASM release 11.2 only, because Oracle ASM is in the grid infrastructure home, which also includes Oracle Clusterware.

  • Oracle Clusterware release 11.2 supports Oracle Database 11g release 2 (11.2), release 1 (11.1), Oracle Database 10g release 2 (10.2), and release 1 (10.1)

  • Oracle ASM release 11.2 requires Oracle Clusterware release 11.2 and supports Oracle Database 11g release 2 (11.2), release 1 (11.1), Oracle Database 10g release 2 (10.2), and release 1 (10.1)

  • Oracle Database 11g release 2 (11.2) requires Oracle Clusterware 11g release 2 (11.2)

    For example:

    • If you have Oracle Clusterware 11g release 2 (11.2) installed as your clusterware, then you can have an Oracle Database 10g release 1 (10.1) single-instance database running on one node, and separate Oracle Real Application Clusters 10g release 1 (10.1), release 2 (10.2), and Oracle Real Application Clusters 11g release 1 (11.1) databases also running on the cluster. However, you cannot have Oracle Clusterware 10g release 2 (10.2) installed on your cluster, and install Oracle Real Application Clusters 11g. You can install Oracle Database 11g single-instance on a node in an Oracle Clusterware 10g release 2 (10.2) cluster.

    • When using different Oracle ASM and Oracle Database releases, the functionality of each is dependent on the functionality of the earlier software release. Thus, if you install Oracle Clusterware 11g and you later configure Oracle ASM, and you use Oracle Clusterware to support an existing Oracle Database 10g release 10.2.0.3 installation, then the Oracle ASM functionality is equivalent only to that available in the 10.2 release version. Set the compatible attributes of a disk group to the appropriate release of software in use.

      See Also:

      Oracle Database Storage Administrator's Guide for information about compatible attributes of disk groups

• There can be multiple Oracle homes for the Oracle database (both single instance and Oracle RAC) in the cluster. Note that the Oracle RAC databases must be running Oracle9i Database, or higher.

• You can use different users for the Oracle Clusterware and Oracle database homes if they belong to the same primary group.

• As of 11g release 2 (11.2), there can only be one installation of Oracle ASM running in a cluster. Oracle ASM is always the same version as Oracle Clusterware, which must be the same (or higher) release than that of the Oracle database.

• For Oracle RAC running Oracle9i you must run an Oracle9i cluster. For UNIX systems, that is HACMP, Serviceguard, Sun Cluster, or Veritas SF. For Windows and Linux systems, that is the Oracle Cluster Manager. To install Oracle RAC 10g, you must also install Oracle Clusterware.

• You cannot install Oracle9i RAC on an Oracle Database 10g cluster. If you have an Oracle9i RAC cluster, you can add Oracle RAC 10g to the cluster. However, when you install Oracle Clusterware 10g, you can no longer install any new Oracle9i RAC databases.

• Oracle recommends that you do not run different cluster software on the same servers unless they are certified to work together. However, if you are adding Oracle RAC to servers that are part of a cluster, either migrate to Oracle Clusterware or ensure that:

  • The clusterware you run is supported to run with Oracle RAC 11g release 2 (11.2).

  • You have installed the correct options for Oracle Clusterware and the other vendor clusterware to work together.