Professional Documents
Culture Documents
D72641
June 2011
Edition 1.0
D71299GC10
Student Guide
Administration
Oracle Exalogic Elastic Cloud
TJ Palazzolo Disclaimer
Technical Contributors other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered
and Reviewers in any way. Except where your use constitutes "fair use" under copyright law, you
James Bayer may not use, share, download, upload, copy, print, display, perform, reproduce,
publish, license, post, transmit, or distribute this document in whole or in part without
Rick Eppstein the express authorization of Oracle.
Mike Palmeter
The information contained in this document is subject to change without notice. If you
Dave Winter find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Editors
Restricted Rights Notice
Pj Schemenaur
Richard Wallis
Shaik Basha Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names
may be trademarks of their respective owners.
Srividya Rameshkumar
Contents
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
1 Course Overview
Course Objectives 1-2
Target Audience 1-3
Introductions 1-4
Course Schedule 1-5
Course Practices 1-7
3 Hardware Components
Objectives 3-2
Exalogic Machine Topology 3-3
Compute Nodes 3-4
Compute Node: Front View 3-5
Compute Node: Rear View 3-6
iii
Compute Node Memory 3-7
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
iv
Summary 4-25
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
6 Storage Configuration
Objectives 6-2
ZFS Appliance Capabilities 6-3
Administrative Tasks 6-4
Getting Started 6-5
Accessing the Browser User Interface (BUI) 6-6
Viewing Appliance Status 6-7
Viewing Network Settings 6-9
Some Supported Services 6-10
Configuring File System Services 6-11
Identifying Hardware Components 6-12
Replacing Failed Hardware 6-14
Performing Failover Actions 6-15
Creating Administrative Users 6-16
Roles and Authorizations 6-17
Shared Storage Concepts 6-18
Default Storage Configuration 6-19
Creating a Pool 6-20
Pool Data Profiles 6-21
v
Creating a Project 6-22
Creating a Share 6-24
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
7 Network Configuration
Objectives 7-2
Exalogic Networks: Review 7-3
IB Networking Concepts: Review 7-4
Default Compute Node Network 7-5
Default Storage Appliance Network 7-6
Quarter-Rack Network Topology 7-7
Half-Rack Network Topology 7-8
Full-Rack Network Topology 7-9
Multi-Rack Topology 7-10
Transceivers and Cables 7-11
Administration Tasks 7-12
Status LEDs 7-13
Replacing Failed Hardware 7-14
IB Switch Command-Line Interface (CLI) 7-15
Monitoring the Hardware 7-16
Monitoring Connection Status 7-17
Running IB Diagnostics 7-18
Subnet Manager (SM) Concepts 7-19
Subnet Manager Recommendations 7-20
Configuring the Subnet Manager 7-21
Fabric Monitor Interface 7-22
Fabric Monitor Examples 7-23
Monitoring IB Connections 7-24
Monitoring Gateway Connections 7-25
vi
Multiple Addresses Versus Multiple Ports 7-26
Configuring a Virtual Address 7-27
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
vii
Some Channel Protocols 9-11
Creating a Cluster Replication Channel 9-12
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
viii
Summary 10-32
Practice 10-2 Migrate Failed Servers 10-33
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
ix
OHS Architecture 12-17
Oracle Process Manager and Notification Server (OPMN) 12-18
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
x
These eKit materials are to be used ONLY by you for the express purpose SELF STUDY. SHARING THE FILE IS STRICTLY PROHIBITED.
Course Overview
Target Audience
If you are concerned whether your background and experience match that of the target
audience, ask the instructor.
Introduce yourself.
Tell us about:
Your company and role
Any previous experience with Sun hardware
Any previous Fusion Middleware product experience
Day Lesson
1 AM 1. Course Overview
2. The Exalogic Solution
3. Hardware Components
PM 4. Initial Machine Configuration
5: Fusion Middleware Concepts
Course Schedule
The course schedule might vary depending on the pace of the class. The schedule shown is
an estimate. The instructor may provide updates.
Day Lesson
Classroom Guidelines
These guidelines enable you to get maximum benefit from the course.
Topic Website
Education and training http://education.oracle.com
Product documentation http://www.oracle.com/technology/documentation
Product downloads http://www.oracle.com/technology/software
Product articles http://www.oracle.com/technology/pub/articles
Course Title
Sun Fire Server Installation and Administration
Sun Storage 7000 Administration
Oracle WebLogic Server: Administration Essentials
Oracle WebLogic Server: Advanced Administration
Related Training
Note that some of the courses listed in the slide are available as traditional or virtual
classroom training, while others are self-paced online offerings (eStudies).
What Is Exalogic?
Exalogic is an integrated hardware and software system designed to provide a complete
platform for a wide range of application types and widely varied workloads. Exalogic is
intended for large-scale, performance-sensitive, mission-critical application deployments. It
combines Oracle Fusion Middleware and Sun hardware to enable a high degree of isolation
between concurrently deployed applications, which have varied security, reliability, and
performance requirements. Exalogic enables customers to develop a single environment that
can support end-to-end consolidation of their entire applications portfolio.
Exalogic hardware is preassembled and delivered in standard 19 42U rack configurations.
The main hardware components of a single Exalogic machine Full Rack are the following:
30 Sun Fire X4170 M2 compute nodes
One dual controller Sun ZFS Storage 7320 appliance with 20 disks
Four Sun Network QDR InfiniBand Gateway Switches
One Sun Datacenter InfiniBand Switch 36
One 48-port Cisco Catalyst 4948 Ethernet management switch
Two redundant 24 kVA power distribution units
Why Exalogic?
The combination of Oracle Exalogic software and hardware results in substantial performance
gains for Java based applications running on WebLogic Server and other Oracle Fusion
Middleware technologies. In addition to performance, the fact that Exalogic hardware and
software have been engineered together also means that customers are required to do far
less in setting up and running Exalogic than for alternative environments.
InfiniBand is fundamental to the Exalogic Elastic Cloud system. In addition to providing an
extremely fast, high-throughput interconnect between all of the hardware units within a
deployment, it also provides extreme scale, application isolation, and elasticity. Traditional
approaches to growing a data centers compute capacity involve either vertically scaling
individual computers or networking together many individual computers by using common
technologies such as Ethernet. By contrast, the lossless switched InfiniBand I/O fabric (on
which the Exalogic system is based) connects all configurations together, essentially forming
a single large computer.
An Exalogic cloud is a resource pool that can be dynamically subdivided into secure units of
capacity. InfiniBand supports partitions, in which communication between endpoints on the
I/O fabric is strictly controlled. Individual compute nodes, or even specific I/O devices, may be
grouped into logical partitions within which communication is allowed.
What Is InfiniBand?
Moving data between applications over a traditional network can consume a lot of time and
drain precious server resources. With traditional network technologies, data exchanges
traverse the operating systems on both the source and destination servers, resulting in
excessive application latency due to operating system calls, buffer copies, and interrupts.
InfiniBand, which today delivers 40 GB per second connectivity with application-to-application
latency as low as 1 microsecond, has become a dominant fabric for high-performance
enterprise clusters. Its ultra-low latency and near-zero CPU utilization for remote data
transfers make InfiniBand ideal for high-performance clustered applications.
InfiniBand also provides a direct channel from the source application to the destination
application, bypassing the operating systems on both servers. InfiniBands channel
architecture eliminates the need for OS intervention in network and storage communication.
This frees server memory bandwidth and CPU cycles for application processing.
In addition to carrying all InfiniBand traffic, the Sun Network QDR InfiniBand Gateway Switch
enables all InfiniBand attached servers to connect to an Ethernet LAN by using standard
Ethernet semantics. No application modifications are required for applications written to use
standard Ethernet.
Engineered Solution
An engineered system combines network, storage, processing, and software into a pre-
optimized package. An engineered approach enables IT to cut deployment time from months
to hours, reduces errors, and enables IT personnel and resources to be allocated to higher-
value work. It also reduces the overall time and costs associated with planning, procurement,
installation, and testing. Exalogic provides a repeatable model for building out data center
systems.
Java Performance
Exalogic has been designed from the ground up to provide the ideal environment for
enterprise Java applications and Java based infrastructure. Oracles entire Fusion Middleware
portfolio is optimized for deployment on Exalogic. Oracle Exalogic Elastic Cloud software
includes a number of optimizations and enhancements made to the core products within
Oracle WebLogic Suite, which includes Oracle WebLogic Server, Oracle Coherence, Oracle
JRockit, and Oracle HotSpot. The combination of Oracle Exalogic software and hardware
results in substantial performance gains for Java based applications running on WebLogic
Server and other Oracle Fusion Middleware technologies.
High Availability
Hardware components within an Exalogic machine use multiple physical InfiniBand (IB) ports
and switches for high availability. Each InfiniBand adapter has two ports, is assigned a single
IP address, and supports active/passive bonding. The active port of the adapter connects to
an IB switch and the passive port connects to another switch. If one fails, traffic is
transparently failed over to the other.
Clustering is configuring a group of WebLogic Servers to work together to provide client
access to the services offered by the servers in the cluster. The cluster appears to a client as
one instance, whether the client is a Web client or a Java application. By replicating the
services provided by one instance, an enterprise system achieves a fail-safe and scalable
environment. Scalability is achieved by balancing the load of incoming requests across the
servers in the cluster.
When a migratable WebLogic Server becomes unavailable for any reason (for example, if it
hangs or loses network connectivity, or if its host machine fails), migration is automatic. Upon
failure, the server is automatically restarted on the same machine if possible. If the server
cannot be restarted on the machine where it failed, it is migrated to another machine. In
addition, an administrator can manually initiate migration of a server instance.
Multi-Tenancy
By default, the Exalogic machine includes a single InfiniBand partition at the rack level. All
Exalogic compute nodes and the storage appliance are full members of this default partition.
The Exalogic network hardware allows you to configure multiple partitions to restrict traffic
between specific IB connections. However, Oracle currently only supports the use of the
default partition to help ensure compatibility between multiple Exalogic and/or Exadata racks.
The Exalogic network hardware does support virtual LANs (VLANs) to achieve a similar result
to partitions. Finally, you can limit access to the network by using multiple subnet masks. The
subnet masks will be enforced at the OS level, not by the network hardware.
If you are using the Solaris OS on Exalogic, you can also utilize the zones feature. Zones act
as completely isolated, virtual operating systems within a single OS instance. Each zone has
its own node name, virtual network interfaces, and storage assigned to it.
You can limit access to an Exalogic shared file system by using both standard UNIX file
permissions and IP- or DNS-based client exceptions. When a client attempts access, its
access will be granted according to the first exception in the list that matches the client (or, if
no such exception exists, according to the global share mode).
Software Compatibility
In addition to unique support for Java applications and Oracle Fusion Middleware, Exalogic
also provides users with a choice of Oracle Linux or Oracle Solaris operating systems.
Exalogic is 100% compatible with all standard Oracle Linux 5 and Solaris 11 applications, and
no special certification for Exalogic is required. In other words, all Oracle applications that are
certified for the appropriate releases of Oracle Linux and Solaris are supported on Exalogic.
The Exalogic specific optimizations for Fusion Middleware are certified only against Exalogic.
They are not supported on other non-Exalogic hardware, and regardless of whether the
hardware utilizes InfiniBand. Currently, Exalogic is not certified to run virtualization platforms
such as Oracle VM or VMware, although this is expected to change in the future.
At the factory:
Both Linux and Solaris base images are pre-installed on the
storage appliance
The compute nodes are configured to run Linux by default
The latest base images are also available on
edelivery.oracle.com.
Hardware:
Exalogic racks and supporting hardware are sold in one of
three specific configurations, depending on initial capacity
needs.
To provide quality support, racks can be expanded and
combined only in specific ways.
Answer: d
Answer: a, c
Hardware Components
Clients
Rack
Compute Node
InfiniBand Mgmt 1 Gb
Storage Appliance
Switches Switch
40 Gb
IB
Network
Power Distribution Units (PDUs)
Dual processor
Redundant power and cooling
Redundant solid state boot disks
InfiniBand host channel adapter
Compute Nodes
Data centers often standardize on a server that can run a wide range of business applications
and can be managed easily. The versatility of the Sun Fire X4170 M2 server makes it a
perfect fit for this, because this server balances compute power, memory capacity, and I/O
capability in a compact and energy-efficient, 1-rack unit (RU) enclosure. This is a server that
is ideal for middleware workloads, system administration, and application development.
The Sun Fire X4170 M2 server includes two Intel Xeon Processor 5600 series, each with six
processing cores. This processor includes an energy management feature that reduces
power consumption when workload decreases. In addition, the server has built-in hardware
and software power management capabilities to further enhance power efficiency.
Each compute node contains two solid-state disks (SSDs), which host the operating system
images used to boot the node, act as high-performance local swap space, and also provide
storage for any diagnostic data generated by the system during a failure. Mirroring for the two
solid-state disk drives is accomplished via a RAID controller.
The Sun Fire X4170 M2 server supports up to 18 double data rate (DDR3) memory dual inline
memory module (DIMMs) slots, which can be populated with 4 GB and 8 GB DIMMs.
However, within an Exalogic configuration, Oracle supports only the twelve 8 GB DIMMs that
are installed at the factory.
32 GB SATA Solid
State Disk (2) USB (2)
Power
Button
Exalogic Storage
The Exalogic ZFS Storage 7320 system consists of two storage controllers or heads in a
high-availability cluster configuration, along with one disk shelf. The 7320 controller base
configuration includes two CPUs, 4 Gb Ethernet ports, redundant power supplies, an
InfiniBand network controller, and dual-port SAS HBA for storage expansion. The CPUs are
Intel Xeon 5600 series, 2.40 GHz, and contain four core processors each (a total of eight
cores). The clustered configuration simply uses two servers and a cluster card in each server
for a heartbeat connection between them.
All user-accessible storage is provided by one Sun disk shelf that is external to the two
servers. Solid-state drives are used as a high-performance write cache known as LogZilla or
ZFS intent log (ZIL) devices, and are in place of 4 of the 24 drives in the disk shelf. The
remaining 20 drives are available for storage.
The RAID function is performed by the software. RAID-1 is recommended, but other options
are available. In the default configuration, there are twenty 2 TB disks (40 TB in total space)
but two of them are used as spares. Therefore, there is 36 TB total disk space or 18 TB of
replicated disk space.
Storage Node
Compute File write
Write Cache/Buffer
Disk Array
Disk Caching
The Exalogic storage appliance transparently executes writes to low-latency SSD media so
that writes can be quickly acknowledged, allowing the application to continue processing.
Then, it automatically flushes the data to high-capacity drives as a background task.
Another type of SSD media acts as a cache to reduce read latency, and the appliance also
transparently manages the process of copying frequently accessed data into this cache to
seamlessly satisfy read requests from clients.
2 TB Hard Disk
Drive (20)
InfiniBand Switches
Within Exalogic, the InfiniBand gateway switches provide a shared, high-speed network for
application clusters that comprise compute nodes and the shared storage appliance. The
gateway switch features a standard 1U data center form factor, thirty-two 40 Gb InfiniBand
ports, eight 10 Gb Ethernet ports, and the Sun ILOM management interface.
Oracle's converged fabric leverages the properties of the InfiniBand architecture to enable
high performance in demanding, clustered data center environments. The fabric supports the
creation of logically isolated network partitions as well as advanced features for traffic
isolation and quality of service (QoS) management. QDR InfiniBand delivers 40 Gb of raw
bandwidth. The InfiniBand base rate is 10 Gb per second. QDR-capable products deliver four
times the base rate.
Deploying a converged fabric minimizes the cost and complexity of building and operating
scalable systems by reducing the number of adapters, cables, and switches to install,
manage, and maintain. LAN traffic carried over Oracle's converged data center fabric uses
familiar Ethernet semantics. Furthermore, the LAN ports on the Sun Network QDR InfiniBand
Gateway Switch appear as endpoints to the LAN, thus avoiding potential interoperability
issues with network management tools and methods.
Redundant Fans
(3; 5 max.)
Management Switch
The Cisco Catalyst 4948 offers 48 ports of wire-speed 10/100/1000BASE-T with four
alternative wired ports that can accommodate optional 1000BASE-X Small Form-Factor
Pluggable (SFP) optics. Exceptional reliability and serviceability are delivered with optional
internal AC or DC 1+1 hot-swappable power supplies and a hot-swappable fan tray with
redundant fans.
The Cisco Catalyst 4948 includes a single, dedicated 10/100 console port and a single,
dedicated 10/100 management port for offline disaster recovery. Remote in-band
management is available with the Simple Network Management Protocol (SNMP), telnet
client, Bootstrap Protocol (BOOTP), and Trivial File Transfer Protocol (TFTP). Support for
local or remote out-of-band management is delivered through a terminal or modem attached
to the console interface. The management port helps enable the Cisco Catalyst 4948 to
reload a new image from a TFTP server.
Multi-Rack
Full Rack
Quarter Half Rack
Rack
Power Options
To protect your Exalogic machine from such disturbances, you should have a dedicated
power distribution system, power-conditioning equipment, as well as lightning arresters or
power cables to protect from electrical storms. The cabinets for the Exalogic machine are
shipped with grounding-type power cords (three-wire). Always connect the cords to grounded
power outlets. For additional grounding, attach a chassis earth ground cable to the Exalogic
machine. The additional ground point enables electrical current leakage to dissipate more
efficiently.
Each rack has two pre-installed power distribution units (PDUs). Different types of PDUs
accept different types of power sources and use different numbers of power outlets. You must
specify the type of PDU that is correct for your data center.
If you upgrade a quarter rack or half rack that was originally ordered with a 15-kVA PDU to a
full rack, you must also upgrade to a 24-kVA PDU.
Answer: b, d, e
Answer: a, c
Answer: d
Answer: c
Power-On Procedure
For a newly installed Exalogic rack, unfasten the power cord cable ties. The ties are for
shipping only and are no longer needed. Then route the power cords to the facility receptacles
either above the rack or below the flooring, and secure the power cords in bundles. Lastly,
plug the PDU power cord connectors into the facility receptacles. The 24 kVA PDU has four
plugs, and the 15 kVA PDU has two plugs
Switch on the six power distribution unit (PDU) circuit breakers located on the rear of the main
PDU (PDU A) inside the Exalogic machine. The circuit breakers are on the rear of the
Exalogic machine cabinet. Press the ON (|) side of the toggle switch. Wait three to five
minutes for all Oracle Integrated Lights Out Manager (ILOM) service processors to boot.
Next, open the front cabinet door. Verify that server standby power is on for compute nodes,
InfiniBand gateway switches, and storage appliance in the Exalogic machine. When standby
power is distributed to the chassis, the service processor (SP) powers on and starts up to
manage the system. The main power is supplied for the remainder of the chassis and fans
when the power button on the front of the server chassis is pressed. The power button is
disabled while the SP is starting. After the SP has started, the power button is enabled and
the system is placed in standby power mode.
1 2 3
Site Requirements
The operating temperature range is 15 degrees Celsius to 32 degrees Celsius (59 degrees
Fahrenheit to 89.6 degrees Fahrenheit). The humidity range is 10 percent to 90 percent
relative humidity, noncondensing.
The minimum ceiling height for the cabinet is 2300 mm (90 inches), measured from the true
floor or raised floor, whichever is higher. An additional 914 mm (36 inches) is for top
clearance. The space above the cabinet and its surroundings must not restrict the movement
of cool air between the air conditioner and the cabinet, or the movement of hot air coming out
of the top of the cabinet. Oracle recommends that the Exalogic machine be installed on raised
flooring. The site floor and the raised flooring must be able to support the total weight of the
Exalogic machine.
The cabinets for the Exalogic machine are shipped with grounding-type power cords (three-
wire). Always connect the cords to grounded power outlets. Because different grounding
methods are used, depending on location, check the grounding type, and refer to
documentation such as IEC documents for the correct grounding method.
Always provide adequate space in front of and behind the rack to allow for proper ventilation.
Do not obstruct the front or rear of the rack with other equipment or objects.
Exalogic Networks
Exalogic machine includes compute nodes and the Sun ZFS Storage 7320 appliance, as well
as equipment to connect the compute nodes to your network. The network connections allow
the servers to be administered remotely, enable clients to connect to the compute nodes, and
enable client access to the storage appliance. Additional configuration, such as defining
multiple virtual local area networks (VLANs) or enabling routing, may be required for the
switches to operate properly in your environment. This type of configuration is beyond the
scope of the installation service.
There are up to five networks for Exalogic machine. Each network must be on a distinct and
separate subnet from the others. The Exalogic management network connects to your
existing management network, and is used for administrative work for all components of
Exalogic machine. It connects ILOM, compute nodes, server heads in the storage appliance,
and switches connected to the Ethernet switch in the Exalogic machine rack. This
management network is in a single subnet. Do not use the management network interface
(ETH0/NET0) on compute nodes for client or application network traffic. Cabling or
configuration changes to these interfaces on Exalogic compute nodes is not permitted.
Worksheet Inputs
Host names are derived from the assigned rack name (el01,
el02, el03, by default).
Default Hostnames
The Exalogic machine name is used to derive the host names for the network interfaces of all
components. For example, an Exalogic machine name of el01 will result in compute node
host names of el01cn01, el01cn02, el01cn03, and so on. In a multi-rack configuration, each
Exalogic machine name should be unique and identify the rack. Oracle recommends using
el01 for the first Exalogic rack, el02 for the second, el03 for the third, and so on.
Private IPs
Client IPs
Installation Spreadsheet
The Exalogic installation spreadsheet is divided into a Configuration tab and a Preview tab.
Entries made on the Configuration tab prepopulate entries on the Preview tab. Entries on the
Preview tab are used as the basis for the configuration that will be generated and used with
ECU. You can also manually edit the Preview values for any fine tuning, although this is not
recommended.
Enter configuration parameters in the General section of the spreadsheet, including the
machine name, and base names for storage and compute nodes. These values are used as
prefixes for host names that will be generated. Enter the domain name, region, and time zone,
as applicable. The Customer Name field determines the location to which the configuration
file will be saved.
Enter configuration parameters in the Network section of the spreadsheet. In this section, you
enter the list of DNS Name Servers, NTP Servers, Search Domains, and Default Gateway.
The start IP Address (Network), Subnet Mask, and Gateway will be the same across all
NET0, ILOM, BOND0, and BOND1 interfaces. As you enter values in these fields, the
corresponding fields on the Preview tab are populated with values.
ECU Internals
If you are running ECU for the first time, it generates a file representing the factory
configuration of the rack. Review the exalogic_current.conf and
exalogic_pending.conf files to ensure that they are accurate. Verify that the current
configuration has the correct number of nodes and switches and the numbers match with the
pending configuration. Verify IP addresses, host names, and such network parameters in both
files.
ECU verifies ILOM for all compute nodes listed in the exalogic_current.conf file. The
master node is omitted from this process because it cannot ping its own ILOM when sideband
management is used. ECU also verifies entries corresponding to the storage heads in the
exalogic_current.conf file and validates that their corresponding ILOMs can be pinged
as well.
ECU verifies that each Sun Network QDR InfiniBand Gateway Switch (NM2-GW) is running a
Subnet Manager (SM). If a switch is not running an SM, an SM is dynamically started on that
switch. It also runs the verify-topology utility, which validates the network cabling.
ECU distributes keys to each compute node and allows subsequent steps to operate more
efficiently by not prompting for a password. After this step runs, accessing remote compute
nodes via SSH will not require a password.
SDP Socket Direct Protocol Applications communicate directly with the IB fabric,
bypassing the operating system's TCP/IP stack.
IPoIB, SDP
bond0
Switch1 IB0
Apps
EoIB vNIC0
Switch2 IB1
bond1
vNIC1
Compute Node
Bonded Interfaces
The Exalogic compute nodes and storage appliance include a Host Channel Adapter (HCA)
to communicate with an InfiniBand network. The HCA consists of two ports, IB0 and IB1.
These raw IB connections are represented on the operating system as the interfaces ib0 and
ib1. The operating system and applications are generally not intended to use these interfaces
directly.
InfiniBand switches (both leaf and spine switches) are configured to automatically separate
the IPoIB traffic and the EoIB traffic. After the InfiniBand gateway switches are connected to
the Exalogic compute nodes, the following bonded interfaces are defined:
bond0 for IPoIB traffic and for using SDP
bond1 for EoIB traffic (uses two vNICs to communicate through ib0 and ib1)
Each HCA has an IP address, and active/passive bonding is configured. The active port of the
HCA connects to an IB switch and the passive port of the HCA connects to another switch. If
one connection fails, traffic is transparently failed over to the other.
Solaris IPMP provides the same functionality as bonded Interfaces on Oracle Linux. Oracle
recommends that you name your IPMP groups bond0 and bond1 to keep the terminology
consistent with Oracle Linux.
Private IB
Network
Client Ethernet
Network
Mgmt Mgmt
Network Switch
# check_ibports
...
Port 1 is in the ACTIVE state
# verify-topology t quarterrack
Quarter rack expected to have 2 internal switches but has 1
... [FAILURE]
# CheckSWProfile
[SUCCESS]....Has supported operating system
[SUCCCES]....Has supported processor
[SUCCESS]....Kernel is at the supported version
[SUCCESS]....Has supported kernel architecture
[SUCCESS]....Software is at the supported profile
Utility Examples
The check_ibports tool determines which of the two ports on the local IB Host Channel
Adapter (HCA) is active.
The verify-topology tool validates the expected cabling and connectivity for a full, half, or
quarter rack configuration.
The CheckSWProfile tool validates the current compute nodes operating system, kernel,
software packages, and hardware architecture for supported levels.
DCLI Examples
To configure SSH user equivalence between the host from which DCLI will be run and a
target host, use the setup-ssh utility, which is found in the same location as DCLI.
To give DCLI a list of hosts on which to run the specified command, supply a comma-
separated list with the -c option or supply the location of a file that lists the host names with
the -g option. Then either give a command to run, surrounded by quotation marks, or provide
the name of a script file with the x argument. The script file will be uploaded to the users
home folder on each target host before being executed.
Answer: b
Answer: b
Answer: c
Middleware:
Extends the traditional client/server software architecture
Is the software that sits between clients, business
applications, and business data
Should be invisible to users
Is a key component of distributed enterprise systems
Middleware Concepts
Middleware is the software that connects software components or enterprise applications.
Middleware is the software layer that lies between the operating system and the applications
on each side of a distributed computer network. Typically, it supports complex, distributed
business software applications. It is especially integral to information technology that is based
on web services and service-oriented architecture (SOA). Middleware makes application
development easier by providing common programming abstractions, by masking application
heterogeneity and the distribution of the underlying hardware and operating systems, and by
hiding low-level programming details.
Because of the continued growth and use of network-based applications by businesses,
middleware technologies have become increasingly important. Companies and organizations
are now building enterprise-wide information systems by integrating previously independent
applications with new software developments. Middleware can provide uniform, standard, and
high-level interfaces to application developers and integrators so that applications can easily
be composed, reused, ported, and made to interoperate.
OS/Hardware
WebLogic Server:
Is a Java application
Hosts Java EE applications
Can be administered from a web-based console or a
scripting interface
Provides clustering services for load distribution and high
Web
Application
Server
EJB
Application
Server Server
Product Product
Installation Product Installation Installation
config
servers
Domain
Machine 1 Machine 2
Server Server
Server Server
Node Manager:
Is a process that accepts remote commands to start, stop,
or suspend servers on the same machine
Monitors server availability and restarts failed ones
Can be used to migrate servers on a failed machine to
another machine
Server
TLOGs
Transaction Recovery
WebLogic Server supports distributed transactions and the two-phase commit protocol for
enterprise applications. A distributed transaction is a transaction that updates multiple
resource managers (such as databases) in a coordinated manner. In contrast, a local
transaction begins and commits the transaction to a single resource manager that internally
coordinates API calls; there is no transaction manager. The two-phase commit protocol is a
method of coordinating a single transaction across two or more resource managers. It
guarantees data integrity by ensuring that transactional updates are committed in all of the
participating databases, or are fully rolled back out of all the databases, reverting to the state
prior to the start of the transaction. In other words, either all the participating databases are
updated, or none are updated.
WLS executes a recovery procedure when failures occur. It determines which transactions
were active in the machine at the time of the crash, and then determines whether the
transaction should be rolled back or committed. Because this service is designed to gracefully
handle transaction recovery after a crash, Oracle recommends that you attempt to restart a
crashed server and allow the service to handle incomplete transactions. When moving
transaction log records after a server failure, make all transaction log records available on the
new machine before starting the server there.
Server
Producer Message Server Consumer
Store
Message Recovery
An enterprise messaging system enables applications to communicate with one another
through the exchange of messages. A message is a request, report, and/or event that
contains information needed to coordinate communication between different applications.
Java Message Service (JMS) servers reside on a WebLogic Server instance, host a defined
set of JMS modules, and also define any associated persistent storage for recovery.
A persistent message is guaranteed to be delivered once-and-only-once. The message
cannot be lost due to a JMS provider failure and it must not be delivered twice. It is not
considered sent until it has been safely written to a file or database. Non-persistent messages
are not stored. They are guaranteed to be delivered at-most-once, unless there is a JMS
provider failure, in which case messages may be lost, and must not be delivered twice.
For storing persistent messages, you can simply use the host server's default persistent file
store, which requires no configuration on your part. However, you can also create a dedicated
file-based store or JDBC store for JMS. A file store maintains subsystem data, such as
persistent JMS messages and durable subscribers, in a group of files in a directory. A JDBC
store maintains subsystem data in a relational database.
Domain
Server 1
Web client Proxy
Server 2
EJB client
JMS client Server n
Compute Node A
Admin Server Domain
Server 1 Server 2 Server 3
Admin
Cluster Server Node
Manager
Answer: b
Answer: d
Answer: c, e
Storage Configuration
Administrative Tasks
Analytics provides real-time graphs of various statistics, which can be saved for later viewing.
About a dozen high-level statistics are provided, such as NFSv3 operations/sec, which can
then be customized to provide lower-level details. Groups of viewed statistics can be saved as
worksheets for future reference.
Getting Started
All stand-alone controllers should have at least one NIC port configured as a management
interface. Select the Allow Admin option in the browser user interface (BUI) to enable BUI
connections on port 215 and CLI connections on SSH port 22. Gather the following
information in preparation for configuring an Ethernet interface on the storage controller: IP
address, IP netmask, host name, DNS domain name, DNS server IP address, and gateway IP
address.
You can access the system by connecting an administrative client to the serial management
port on the controller. Alternatively, if a Dynamic Host Configuration Protocol (DHCP) server is
on the network, you can access the system by connecting the network management port to
your network. For a serial connection, use telnet. For an Ethernet connection, use SSH. At the
command prompt, type start /SP/console. After you configure the primary interface,
configure the remaining system parameters by using the BUI from any client on the same
network.
Note that if you are installing a cluster, configure only one controller initially. The software
propagates the configuration to the peer controller during cluster initialization. After the cluster
is initialized, you can administer the system from either storage controller. However, do not
attempt initial configuration on both controllers independently.
Power off or
reboot appliance.
Select a pool. 1
2
Service
status
Disk usage
Memory
Hardware
availability
Click to
view full log.
1
2
3
Ethernet
port status
View/edit addresses
IB port status
1
2
4
Configure
service settings.
1
2 Power off
or restart.
3
el01stor01
Location of selected
component
Change
perspective.
View specifications.
3
Active head Passive head
Cluster ports
are connected
el01stor01 el01stor02
1
2
3
4
Select
Add authorizations
to role
Term Definition
Project Default administrative and file system settings for a collection of shares
Share A file system mount point, access protocols, access rights, and other
settings
Project Project
common
1
2
el01stor01
Creating a Pool
Each node can have any number of pools, and each pool can be assigned ownership
independently in a cluster. While an arbitrary number of pools is supported, creating multiple
pools with the same redundancy characteristics owned by the same cluster head is not
advised. Doing so will result in poor performance, suboptimal allocation of resources, artificial
partitioning of storage, and additional administrative complexity.
When allocating raw storage to pools, keep in mind that filling pools completely results in
significantly reduced performance, especially when writing to shares. These effects typically
become noticeable after the pool exceeds 80% full, and can be significant when the pool
exceeds 90% full. Therefore, best results are obtained by over-provisioning by approximately
20%.
After the task of creating a new pool is started, the setup falls into two different phases:
verification and configuration. The verification phase enables you to verify that all storage is
attached and functioning, and allocate disks within chassis. In a stand-alone system, this
presents a list of all available storage and drive types, with the ability to change the number of
disks to allocate to the new pool. By default, the maximum number of disks is allocated, but
this number can be reduced in anticipation of creating multiple pools.
Profile Description
Striped No redundancy
Single-Parity RAID For every 3 disks, there is a single parity disk for recovery.
Reduces usable space by 1/4
Double-Parity RAID For every 3 disks, there are 2 parity disks for recovery.
Reduces usable space by 2/5
Triple-Parity RAID For every n disks, there are 3 parity disks for recovery.
1 5
2
6
6
Base path for share
3 mount points
Creating a Project
1. Click Shares in the top menu.
2. Click Projects on the submenu.
3. A list of all projects is displayed. Click the plus (+) button to create a new project.
4. Supply the project with a name. By default, this name is used as the base mount point
for this project.
5. Click the General tab for this new project.
6. Configure the default settings for any shares that will be members of this project
including:
- Quota: Sets a maximum limit on the total amount of space consumed by all file
systems and within the project.
- Read-only: The contents of a read-only file system cannot be modified, regardless
of any protocol settings. This setting does not affect the ability to rename, destroy,
or change properties of the file system.
1 2 4
Creating a Share
Shares that are part of a project can either have local settings for properties, or they can
inherit their settings from the parent project. By default, shares inherit all properties from the
project. If a property is changed on a project, all shares that inherit that property are updated
to reflect the new value. When inherited, all properties have the same value as the parent
project, with the exception of the mount point. When inherited, the mount point is the
concatenation of the project setting and the share name.
1. Access a project.
2. Click the Shares menu option.
3. Click the plus (+) button to define a new shared file system.
4. Give the share a name. By default, this name is used as part of the mount point. Edit
other share settings such as:
- Data Migration Source: If set, this file system is actively shadowing an existing
file system, either locally or over NFS.
- Case sensitivity: It controls whether directory lookups are case-sensitive or not
case-sensitive.
1 2
Clear to override
project values.
Setting Quotas
The appliance leverages a pooled storage model where all file systems share common space.
By default, file systems never have an explicit size assigned to them, and only take up as
much space as they need. This approach provides maximum flexibility and simplicity of
management in an environment when users are generally trusted to do the right thing. A
stricter environment, where user's data usage is monitored and/or restricted, requires more
careful management. A quota represents a limit on the amount of space that can be
consumed by any particular entity. This can be based on file system, project, user, or group,
and is independent of any current space usage. Clients attempting to write new data will get
an error message when the file system is full, either because of a quota or because the
storage pool is out of space.
A reservation, on the other hand, represents a guarantee of space for a particular project or
file system. This takes available space away from the rest of the pool without increasing the
actual space consumed by the file system. This setting cannot be applied to users and
groups. The traditional notion of a statically sized file system can be created by setting a
quota and reservation to the same value. By default, a reservation includes all snapshots of a
file system. If, instead, the Include Snapshots property is not set, then the reservation only
applies to the immediate data of the file system.
Starting IP and
CIDR bit mask
Configuration Management
Each update may be supplied with new firmware or updates to external resources. In general,
these updates are backward-compatible and applied automatically without user intervention.
There are exceptions, however, for nonreversible updates. These updates involve updating a
resource external to the system software in a way that is incompatible with older software
releases. After the update is applied, rolling back to previous versions will result in undefined
behavior. For these updates, you will always be given an explicit option to apply them
automatically during upgrade or apply them after the fact.
Following the application of a software upgrade, any hardware for which the upgrade includes
newer versions of firmware will be upgraded. There are several types of devices for which
firmware upgrades may be made available; each has distinct characteristics.
The appliance can generate support bundles containing system configuration information and
core files for use by remote support in debugging system failures. Support bundles are
generated automatically in response to faults if the Phone Home service is enabled.
Administrators can also manually generate and upload a support bundle. To facilitate this, the
appliance must be connected to the Internet, either directly or through the web proxy
configured on the Phone Home service screen.
/etc/fstab (Linux):
1
/etc/vfstab (Solaris):
Recommended settings
2
> mount /u01/apps
users activity
ShareA
roles alerts
Command Description
shares
project CService1
set mountpoint=/export/CService1
set compression=gzip
commit
select CService1
status
storage
show
hardware
show
CLI Examples
To return to the previous context, use the done command. Similarly, to navigate to a parent
context, use the cd command. Inspired by the classic UNIX command, cd takes an argument
of .. to denote moving to the parent context.
exalogic
FMW11gR1 HR
domains
Middleware
recovery
/jms
/tlog
apps
Answer: d
Answer: a
Answer: a, b, e
Network Configuration
SDP Socket Direct Protocol Applications communicate directly with the IB fabric,
bypassing the operating system's TCP/IP stack.
Private IB
Network
ILOM Ethernet
Client
Network
Mgmt Mgmt
Network Switch
Private IB
Network
NFS
Storage Heads
IPoIB IB0
IB
Switches IB1
Mgmt Mgmt
Network Switch
10 Gb Network
1-8 1-8
Redundant links 16 unused
10 Gb IB ports
Redundant
switches
6
NM2-GW #1 NM2-GW #2
2 Storage 2
Heads 12
IB0 (active) IB1 (passive)
8 Compute 8
Nodes 18
10 Gb Network
1-8 1-8
8 unused
IB ports
6
NM2-GW #1 NM2-GW #2
10 Gb Network
1-8 1-8 1-8 1-8
4 6 4
NM2-GW #3 NM2-GW #1 NM2-GW #2 NM2-GW #4
2 Storage 2
Heads 1-2
All four switches No unused
Six unused
NM2-36 Full Rack IB ports
4
Four unused
IB ports
NM2-GW NM2-GW
6 6 NM2-GW NM2-GW
NM2-GW NM2-GW
4
NM2-36
Full Rack
Multi-Rack Topology
Sun Network QDR InfiniBand Gateway Switch (NM2-GW) 4x4 (four switches in each Exalogic
machine full rack) cross-connects are re-cabled to their respective local spine switches (Sun
Datacenter InfiniBand Switch 36 (NM2-36). In addition, each leaf switch is connected to the
other racks spine switch. For example, leaf switches in rack 1 are connected to the spine
switch in rack 2.
Before you can set up the multi-rack cabling, you must first shut down the affected Exalogic
machines and Oracle Exadata Database Machines. Consider the operational impact of
shutting down the systems, and take appropriate action. Also be careful when you handle
InfiniBand cables. InfiniBand cables are very fragile. The cable ends might break off if the
ends are dropped or mishandled in any way.
A spare cable bundle is provided with Exalogic machines. Extract the spare InfiniBand cables
from the bundle. You need to use the InfiniBand cables only. Restore spare Ethernet cables
to the original spares location. You do not use the Ethernet cables.
Re-label all inter-rack cables from leaf switches to spine switches. Re-label each cable to
identify the correct locations at both cable ends, so that cables can be connected correctly
during future maintenance.
Refer to the Exalogic Machine Multirack Cabling Guide for more detailed instructions.
Front
DC power OK
Power supply fault
AC power OK
Fan fault
Switch fault
Switch OK
Rear
Status LEDs
Status LEDs are used on many components of the switch chassis as a means of indicating
the components state. The power supply status LEDs and fan status LEDs are located on the
front of the switch chassis:
The power OK LED is green. When it is on, a 12V DC voltage is present.
The power attention/fault LED is amber. When it is on, a fault has been detected on the
power supply and power has been shut down.
The power AC OK LED is green. When it is on, AC power is present.
The fan attention/fault LED is amber. When it is on, there is a problem with the fan.
The chassis status LEDs, network management status LEDs, and link status LEDs are
located on the rear of the switch chassis:
The switch locator LED is white. When it is flashing, the switch is indentifying itself. The
locator LED can be enabled remotely using the browser or command-line management
interfaces.
The switch attention/fault LED is amber. When it is on, a networking fault has been
detected on the switch.
The switch OK LED is green. When it is off, the switch is off or is initializing. When it is
on, the gateway is functional and no faults are present.
# help all
# help diag
# version
SUN DCS gw version: 1.1.2-2
Build time: Sep 24 2010 10:32:29
FPGA version: 0x33
SP board info:
Manufacturing Date: 2010.06.03
Serial Number: "NCD5A0025"
...
# env_test
...
Voltage test returned OK
...
Temperature test returned OK
...
FAN test returned OK
...
# ibdiagnet
Discovering ... 13 nodes (1 Switches & 12 CA-s) discovered.
...
No bad Links (with logical state = INIT) were found
...
Subnet: IPv4 PKey:0x0001 QKey:0x00000b1b MTU:2048Byte ...
No members found for group
...
# ibnetdiscover
Running IB Diagnostics
The ibdiagnet command performs a collection of tests on the InfiniBand fabric by using
directed route packets and extracts all the available information regarding connectivity. The
command also checks for duplicate node/port GUIDs in the InfiniBand fabric along with
suspected bad links, and displays any issues to standard output. It also generates several
output files, including:
ibdiagnet.lst: List of all the nodes, ports, and links in the fabric
ibdiagnet.fdbs: Dump of the unicast forwarding tables of the fabric switches
ibdiagnet.sm: List of all the Subnet Manager (state and priority) in the fabric
The ibnetdiscover command discovers the InfiniBand network topology and generates a
human-readable topology file. It displays all nodes, node types, node descriptions, links, port
numbers, port LIDs, and GUIDs.
The ibcheckerrors command uses the topology file that was previously generated by
ibnetdiscover to scan the InfiniBand fabric, validate the connectivity as described in the
topology file, and report errors as indicated by the port counters.
Switch A Switch B
SM Master SM
Switch A
SM
# disablesm
# enablesm
# getmaster
Local SM enabled and running ...
# setsmpriority list
smpriority 5 ...
# setsmpriority 8
Monitor the
hardware.
1
Active link LED status
Not connected with errors
2
Monitoring IB Connections
The rear panel diagram displays the presence of connectors and their status by using a
graphic. The diagram displays the management controllers IP address, the connector
receptacles, and their respective connector names. When a cable is attached to a receptacle,
a connection is made. That connection is displayed in the diagram as a gray rectangle, with
three or four smaller indicators. Moving the mouse cursor over an indicator, clicking an
indicator, or clicking a connection opens a window that provides additional information about
that indicator or connection.
In the rear panel diagram, there are 32 InfiniBand receptacles displayed, labeled 0A to 15A
and 0B to 15B. When a connector is physically present in an InfiniBand receptacle, the
receptacle changes from a black rectangle to a gray rectangle with three indicators. Moving
the mouse cursor over an indicator that is orange or red opens a small window that provides
the reason for the respective state. A center indicator is orange when the link is at a speed
slower than QDR, such as SDR or DDR. A right indicator is red when there are significant
errors (symbol, recovery, and so on) on the link.
Clicking a gray InfiniBand connector opens a window that displays connector FRU, port state,
error, and statistical information for that connection.
Machine Machine
192.168.1.1:7003 ServerA OR
192.168.1.1:7003 ServerA
192.168.1.1:7005 ServerB 192.168.1.2:7003 ServerB
A VLAN:
Groups together physical network connections that are
allowed to communicate with one another
Enables remote hosts to communicate as if they were on a
single LAN
Is registered on a specific port of a switch
Switch
Port1: VLAN10
Port2: VLAN10
Port3: VLAN20
Port4: VLAN30
# showvlan
Configuring a VLAN
You can use the showvlan command to identify which VLANs are associated with the IB
gateway 10 Gb Ethernet connectors. The command lists the connectors on the left, and the
associated VLANs and respective partition keys on the right.
When you create a VLAN, you create a mapping between a gateway connector, a VLAN
identifier, and a partition key (0000-7fff). By default on Exalogic only one partition is available,
so only one partition key is valid. You can create VLANs with the createvlan command. If a
VNIC will be associated to a VLAN, you must create the VLAN before the VNIC.
IP subnets:
Enable hosts on the same local network to be isolated
from one another
Are configured and enforced at the OS level by using
subnet masks
Subnet Partitioning
When planning an Exalogic deployment across a collection of compute nodes, use a subnet
mask that provides a suitable IP address range to cover all of these servers on the target
group of compute nodes. In some scenarios, the application from one department may require
communication with the application of another department, each of which is hosted on
separate compute nodes. To enable the applications to communicate, you must set up
another IP subnet in which all target compute nodes are members.
Answer: c
Answer: c
Answer: d
/domainB /domainB
/domainA /domainA
Admin
Server
Server Server
Admin
Server Server Server
Node Node
Node Manager
Manager Manager
Product Installation
Domain 1
FMW Installation 1
Domain 2
FMW Installation 2 Domain 3
./wls1034_linux64.bin
Installation Overview
1. Specify the Oracle Middleware Home directory that will serve as the central directory for
all the Oracle products installed on the target system. You can optionally also enter your
email and account information for online support.
2. A typical Oracle WebLogic Server installation includes all server components, along with
Oracle Coherence if it is part of the installer. It does not include sample code and scripts
for WebLogic and Coherence, however. This option is recommended for production
deployments.
3. In most instances, you can simply accept the default product installation paths, which
are found under the Middleware Home directory you provided earlier.
Oracle also recommends that you make a backup copy of your installation at this stage, and
before the installation of any additional FMW products. However, keep in mind that, with the
exception of product patches, you should not have to modify any of the installation files.
Modifications should be made at the domain level.
cd <WL_HOME>/common/bin
./config.sh
Domain Templates
A domain is the basic administration unit of WebLogic Server. It consists of one or more
WebLogic Server instances, and logically related resources and services that are managed,
collectively, as one unit. In addition to infrastructure components such as servers and
clusters, a domain defines application deployments, supported application services (such as
database and messaging services), security options, and physical host machines.
The Configuration Wizard guides you through the process of creating a domain for your target
environment by selecting the product components that you want to include in your domain, or
by using domain templates. If required, you can also customize the domain to suit your
environment by adding and configuring managed servers, clusters, and machine definitions,
or customizing predefined JDBC data sources, and JMS file store directories.
A domain template defines the full set of resources within a domain, including infrastructure
components, applications, services, security options, and general environment and operating
system options. You can create this type of template from an existing domain by using the
Domain Template Builder, WLST, or the pack command-line tool. Subsequently, you can
create a domain based on the template by using the Configuration Wizard or WLST.
3
5
6
Admin server
settings
Initial list of
managed servers
7
8
9
11
Exalogic
compute nodes
pack.sh domain=domains/test/mydomain
template=templates/mydomain.jar
-template_name="My Custom Template"
unpack.sh template=templates/mydomain.jar
domain=domains/prod/mydomain
username=weblogic
password=mypassword setDomainEnv
commEnv
1 http://<admin_ip>:7001/console
2
4
1
4
Answer: b, c, d
Answer: b, e
IP1:7001
IP3:7001
Primary Secondary
Network Channels
A network channel is a configurable resource that defines the attributes of a network
connection for a specific WebLogic Server instance. A network channel definition includes a
listen address, port number, supported protocol, and whether it can be used for internal
server-to-server communication. You can use network channels to manage quality of service,
meet varying connection requirements, and improve the utilization of your systems and
network resources.
Administrators create a channel for a specific server instance. Channels are not defined
globally and applied to one or more servers. You can assign multiple channels to a server
instance, but each channel must have a unique combination of listen address, listen port, and
protocol. Similarly, if you assign non-SSL and SSL channels to the same server instance,
make sure that they do not use the same port number.
If you want to disable the non-SSL listen port so that the server listens only on the SSL listen
port, deselect Listen Port Enabled in the Configuration > General settings for the server.
Similarly, if you want to disable the SSL listen port so that the server listens only on the non-
SSL listen port, deselect SSL Listen Port Enabled. Note that unless you define custom
network channels, you cannot disable both the default non-SSL listen port and the SSL listen
port. At least one port must be active on a server.
Creating a Channel
To configure a network channel for an existing server, perform the following steps:
1. Select the server, and then click its Protocols tab.
2. Click the Channels subtab.
3. Click New.
4. Enter a name for the channel, select the protocol it will accept or use, and click Next.
For administrative channels, select the admin protocol. For cluster channels, select the
cluster-broadcast or cluster-broadcast-secure protocols.
Value Description
startWebLogic.sh:
...
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true"
Subsystem Description
Answer: a, b, c
Answer: b
Node Manager:
Is a process that accepts remote commands to start, stop,
or suspend servers on the same machine
Monitors server availability and restarts failed ones
Can be used to migrate servers on a failed machine to
another machine
Machine
Server
Node Start/stop
Manager Monitor/
restart Server
exalogic
nodemanager
Pool
Separate folder for
Project
each compute node
Share
Property Description
NodeManagerHome Location of NM property and log files
ListenAddress, Address and port from which NM will accept
ListenPort commands
Authentication Requires a username/password to use this NM
Enabled
java weblogic.WLST
nmEnroll('/u01/HR/domains/MyDomain1',
'/u01/HR/nodemanager/node5')
Trusted NM
The second arg is not required if Domains List Credentials for
running WLST from this location. Domain XYZ
java weblogic.WLST
nmConnect('nmuser','nmpass','myhost','5556','MyDomain',
'/u01/HR/domains/MyDomain')
Connect to an admin server and use it to start other servers, via the NM:
java weblogic.WLST
connect('domainuser','domainpass','myhost:7001')
start('Server2')
start('MyCluster','Cluster')
WebLogic Logs
Node Manager creates a log file located in NodeManagerHome/nodemanager.log by
default. This log file stores data about all of the domains administered by Node Manager. Log
output is appended to the current nodemanager.log. Log rotation is disabled by default, but
can be enabled by setting LogCount and LogLimit in nodemanager.properties.
Node Manager creates the server output log for a server instance in the server instances logs
directory, with the name <server>.out. If the debug property is enabled as a remote start
property for the server instance, or if the Node Manager debug property is enabled, Node
Manager will include additional debug information in the server output log information. You
cannot limit the size of these log files.
Each WebLogic Server maintains its own log file. These log files also support rotation and
filtering, similar to the Node Manager.
Whole-Server Migration
When a migratable server becomes unavailable for any reason (for example, if it hangs or
loses network connectivity, or if its host machine fails), migration is automatic. Upon failure, a
migratable server is automatically restarted on the same machine if possible. If the migratable
server cannot be restarted on the machine where it failed, it is migrated to another machine.
In addition, an administrator can manually initiate migration of a server instance.
Node Manager is used by the Administration Server or a stand-alone Node Manager client to
start and stop migratable servers and is invoked by the cluster master to shut down and
restart migratable servers, as necessary.
Server migration has the following additional requirements:
There is no built-in mechanism for transferring files that a server depends on between
machines. Using a disk that is accessible from all machines is the preferred way to
ensure file availability. If you cannot share disks between servers, you must ensure that
the contents of domain_dir/bin are copied to each machine.
Although migration works when servers are not time synchronized, time-synchronized
servers are recommended in a clustered environment.
Cluster
1
Node Check Machine Status Node
Manager 3 Manager
Server 1 Server 2
Check
4 Add IP 5 Renew
Lease
Server 3
Start Server
Node Node
Server 2
Manager Manager
Type Description
Consensus Servers renew leases by contacting the master directly.
(In-Memory)
Leasing data is maintained in memory on the master and cached
Leasing Types
Using the database leasing implementation, lease information is maintained within a table in a
high-availability database. A high-availability database is required to ensure that leasing
information is always available. Each member of the cluster must be able to connect to the
database to access leasing information. This method of leasing is useful for customers who
already have a high-availability database within their clustered environment. This method also
enables you to use leasing functionality for JMS and Java Transaction API (JTA) service
migration features without also being required to use Node Manager. Database connectivity is
provided through a JDBC data source. Note, however, that XA data sources are not
supported for server migration.
In the nondatabase version of Consensus leasing, WebLogic Server maintains leasing
information in-memory. This removes the requirement of having a high-availability database to
use features that require leasing. One member of a cluster is chosen as the cluster leader and
is responsible for maintaining the leasing information. The cluster leader is chosen based on
the length of time that has passed since startup. The Managed Server that has been running
the longest within a cluster is chosen as the cluster leader. Other cluster members
communicate with this server to determine leasing information. However, the leasing table is
replicated to other nodes of the cluster to provide failover.
...
bond0=10.0.0.1-10.0.0.17,NetMask=255.255.255.224
bond1=10.1.0.1-10.1.0.17,NetMask=255.255.255.224
UseMACBroadcast=true
Leasing Existing
type data source
Candidate Machines
For each server that will support automatic migration to another machine, perform the
following:
1. Select a server in the console.
2. Select Configuration > Migration.
3. Select the Automatic Server Migration Enabled check box and click Save. You must
restart the server for this setting to take effect.
To configure the default list of candidate machines for server migration for a cluster:
1. Select a cluster in the console. Then select Configuration > Migration.
2. Use the Candidate Machines field to limit the available machines for migration. You can
also change the order of preference.
To configure a list of candidate machines to which a specific server in the cluster can be
migrated:
1. Select a server in the console.
2. Select Configuration > Migration.
3. Use the Candidate Machines field to limit the available machines for migration. You can
also change the order of preference.
Machine Failback
When a machine that previously hosted a server instance that was migrated becomes
available again, the reversal of the migration processmigrating the server instance back to
its original host machineis known as failback. WebLogic Server does not automate the
process of failback. An administrator can accomplish failback by manually restoring the server
instance to its original host.
The general procedures for restoring a server to its original host are as follows:
1. Gracefully shut down the new instance of the server.
2. After you have restarted the failed machine, restart Node Manager and the Managed
Server.
1
2
Migrating a Server
To manually migrate a server to another machine, perform the following steps:
1. Select a server in the console.
2. Click the Control tab.
3. Click the Migration second-level tab.
4. In the Migrate to Machine field, select a machine from the list of available candidates.
Then click Save.
exalogic
MyDomain
jms
Pool
Cluster1
Project tlogs
Share
Answer: d
Answer: b
Answer: b, c
Compute Node
Gateway
InfiniBand 10 Gb Ethernet
Application DB
Application
Oracle RAC:
Supports multiple Oracle database servers for greater
scalability
Relies on database servers having access to a shared and
highly available storage device
The Oracle driver provides basic load balancing and
RAC Node 1
Application Driver Shared
RAC Node 2 Storage
Compute Node
InfiniBand
Application DB
Application
Exadata DB Servers
2 1-2 2
Cells 1-3
3 3
7
NM2-36 #1 NM2-36 #2
2 2
NM2-GW #1 NM2-GW #2
6
2 Storage 2
Heads 1-2
8 Compute 8
Exalogic Nodes 1-8
12 unused
IB ports
Exadata
NM2-36 2
4
12 unused NM2-36
Exalogic IB ports
JDBC: Overview
The JDBC API is a natural Java interface for working with SQL. It builds on Open Database
Connectivity (ODBC) rather than starting from the beginning, so programmers familiar with
ODBC find it very easy to learn.
The value of JDBC lies in the fact that an application can access virtually any data source and
run on any platform with a Java Virtual Machine (JVM). That is, with JDBC, you do not have to
write one program to access a Sybase database, another to access an Oracle database,
another to access an IBM DB2 database, and so on. You can write a single program by using
the JDBC API. Because the application is written in Java, you do not need to write different
applications to run on different platforms, such as Windows and Linux.
JDBC accomplishes database connections by using a driver mechanism that translates the
JDBC calls to native database calls. Although most available drivers are fully written in Java
(Type 4) and are thus platform independent, some drivers (Type 2) use native libraries and
are targeted to specific platforms. Oracle WebLogic Server includes several Type 4 JDBC
drivers, which are compliant with the JDBC 3.0 specification.
Data sources:
Enable database connectivity to be managed by the
application server
Use a dynamic pool of reusable database connections
WebLogic Server
RAC Node
Application
TCP or
GridLink Data Source RAC Node
SDP
Application
Driver Connections
RAC Node
ONS
A RAC Node
Application GridLink DS
Service A A
RAC Node
B
Application GridLink DS
Service B B RAC Node
Supports a list or a
Initial list of nodes to single SCAN address
Data Source A
1. DS lookup. Connection
2. Get connection. Connection
App
3. Perform SQL. Connection
Synchronize
App Data Source B
Data Source C
Licensing Review
Active GridLink for RAC is not a binary distribution like WebLogic Server. It is simply the name
given to the use of WebLogic Server's GridLink Data Source feature that includes Fast
Connection Failover and FAN events.
Answer: b
Answer: a
Oracle Coherence:
Is a JVM process
Provides a distributed, in-memory data caching solution
Offers very high performance and scalability
Is based on a cluster of cache servers
Coherence: Overview
One of the primary uses of Oracle Coherence is to cluster an applications objects and data.
In the simplest sense, this means that all the objects and data that an application delegates to
Coherence clusters are automatically available to and accessible by all servers in the
application cluster. None of the objects or data will be lost in the event of server failure. By
clustering the applications objects and data, Coherence solves many of the difficult problems
related to achieving availability, reliability, scalability, performance, serviceability, and
manageability of clustered applications.
The partitioning feature dynamically load-balances data evenly across the entire server
cluster, whereas replication ensures that a desired set of data is always available and up-to-
date at all times in the cluster. Replication enables operations that are running on any server
to obtain the data that they need locally, at basically no cost, because that data has already
been replicated to that server. The only downside of partitioning is that it introduces latency
for data access, and in most applications, the data access rate far outweighs the data
modification rate. To eliminate the latency associated with partitioned data access,
Coherence can use local or near caching" as well. Frequently and recently used data from
the partitioned cache is maintained on the specific servers that are accessing that data, and
this near data is kept up-to-date by using event-based invalidation.
Coherence: Architecture
With Coherence, there is neither master nor slave, but rather all members work together as a
team to manage the cluster as a whole, allowing cluster members to join and leave without
impacting the remainder of the cluster or client applications.
Due to Coherence cluster heartbeat communication, all members are known and it is possible
to provide redundancy within the cluster, such that the death of any one JVM or server
machine does not cause any data to be lost. Also, it is possible to load balance
responsibilities across the cluster. Load balancing automatically occurs to respond to new
members joining the cluster, or existing members leaving the cluster. The death or departure
of a cluster member is automatically and quickly detected, so failover occurs very rapidly, and
more importantly, it occurs transparently, which means that the application does not have to
do any extra work to handle failover.
Coherence servers and clusters can be configured, administered, and started manually
through XML configuration files and custom scripts. Alternatively, Coherence servers and
clusters can be configured as members of a WebLogic domain. The latter approach gives you
the option of modifying basic Coherence settings from the WebLogic console or WLST. You
can also take advantage of the WebLogic Node Manager to remotely start and stop
Coherence servers and to automatically monitor their health.
Applications themselves:
Are members of the Coherence cluster and automatically
discover other members
Can participate in data partitioning (storage-enabled)
Local storage
Application
Coherence Coherence
Server Server
Coherence Cluster
Overrides cluster
defaults
2 3
Specify additional
JVM arguments.
3
1
Start by using
2 Node Manager.
cd <COHERENCE_HOME>
./bin/cache-server.sh
Cache Node
WebLogic Cluster
Cache Node
Coherence*Web: Overview
Coherence*Web is an application server plug-in dedicated to managing session state in
clustered environments. It brings the scalability, availability, reliability, and performance
characteristics of a Coherence data grid to in-memory session management and storage.
Also, because it is not constrained by the deployment topologies of the application server, it
enables session sharing and management across different web applications, domains, and
even different application server products. Sometimes you may want to explicitly prevent
session data from being shared by different Java EE applications that participate in the same
Coherence cluster, so Coherence*Web supports this approach as well.
The Coherence*Web SPI for WebLogic Server is configured with local-storage disabled. This
means a Coherence cache server must be running in its own JVM, separate from the JVM
running WebLogic Server.
With Coherence*Web, session data is stored outside of the application server, thereby freeing
server heap space. This architecture also enables you to individually tune and scale the sizes
of your application server clusters and session data grids.
Domain
Server 1
HTTP
Web client Proxy
Server 2
EJB client
JMS client Server n
Hardware Load
Balancer
Proxy Plug-Ins
An organization's DMZ typically includes an external load balancer or router to handle
external requests from the Internet or your Intranet. The load balancer then forwards the
requests to a web server such as Oracle WebTier or Apache. WebLogic is responsible for
handling the application requests that require dynamic Java EE functionality. The web server
then hosts static content and also, in the case of a cluster, performs load balancing and
failover.
The plug-in does a simple round-robin between all available servers in a cluster. The server
list specified in this property is a starting point for the dynamic server list that the server and
plug-in maintain. WebLogic Server and the plug-in work together to update the server list
automatically with new, failed, and recovered cluster members.
The plug-in also supports sticky session behavior, similar to hardware load balancers. This
means that the plug-in directs HTTP requests that contain a session cookie (or URL-encoded
session ID or a session ID stored in the POST data) to the same server in the cluster that
originally created the cookie. That is, the client is pinned to a specific server in the cluster.
This behavior ensures that users do not lose their session data between requests.
Proxy
Compute Node
Servers
Plug-ins are:
Bundled with WLS and Oracle HTTP Server (OHS)
Also available online to download separately
Released on a separate cycle than that for the server itself,
so always confirm that a newer version is not available
Although all plug-ins share the same major capabilities, the
Product Description
Oracle iPlanet Web Server Supports the WLS plug-in but must be
installed manually
Oracle HTTP Server Based on Apache (httpd.conf)
WLS plug-in is preinstalled (mod_wl_ohs).
Component1 Component2
Client Client
Config, Logs Config, Logs
Instance
OHS Installation
OHS Architecture
Oracle HTTP Server directories are divided between the Oracle home and the Oracle
instance. The Oracle home directories are read-only and contain the Oracle Fusion
Middleware binaries. The Oracle instance directories contain the modules, applications, and
logs for Oracle HTTP Server.
Each OHS component has a root configuration directory found at
<instance>/config/OHS/<component>, which includes the WLS plug-in configuration
file, mod_wl_ohs.conf. Similarly, each components log files are found at
<instance>/diagnostics/logs/OHS/<component>.
When Oracle HTTP Server starts, it writes the process ID (PID) of the parent httpd process to
the httpd.pid file located, by default, in the following directory:
<instance>/diagnostics/logs/OHS/<component>
The process ID can be used by the administrator when restarting and terminating the
daemon. If a process stops abnormally, it is necessary to stop the httpd child processes by
using the kill command. The PidFile directive in httpd.conf specifies the location of the
PID file.
OPMN is:
A process that is used to start, manage, and monitor other
Oracle Web Tier and Identity Management processes
Similar conceptually to the WLS node manager
Accessed by using the opmnctl command-line tool or
Oracle web-based tools
opmnctl startall
Get the name, status, memory usage, and port number of processes:
opmnctl status -l
OPMNCTL: Examples
The opmnctl program provides a centralized way to control and monitor system components
from the command line. The available commands include:
start: Start the OPMN server for a local Oracle instance without starting system
processes.
startall: Start OPMN as well as the system processes for a local Oracle instance.
startall is equivalent to start followed by startproc without arguments.
stopall: Shut down the OPMN server as well as the system processes for the local
Oracle instance. This request operates synchronously. It waits for the operation to
complete before returning.
startproc, restartproc, stopproc: Use these commands to start, restart, or stop system
processes. The OPMN server must be up and running.
The following attributes are supported. Any of these attributes may be omitted, and treated as
a wildcard:
ias-component: The name of a specific managed process, as defined in opmn,xml
process-type: The type of managed process to command, as defined in opmn.xml
process-set: The name of a custom process group, as defined in opmn.xml
Parameter Description
WebLogicHost, Proxy to a single server
WebLogicPort
WebLogicCluster Proxy to this initial list of clustered servers
Latest server list is refreshed with each response
ConnectTimeoutSecs Maximum total wait time to establish a connection
WLSocketTimeout Wait time for a connection attempt
Answer: c
Answer: b, c, e
ILOM:
Is made up of specialized hardware and software for
managing and monitoring Sun devices
Is preinstalled on Exalogic compute nodes, storage nodes,
and InfiniBand switches
Runs independently of any operating systems
ILOM Tasks
The system monitoring features in ILOM enable you to easily determine the health of the
system and to detect errors, at a glance, when they occur. All Oracle Sun server platforms are
equipped with a number of sensors that measure voltages, temperatures, fan speeds, and
other attributes about the system. Each sensor in ILOM contains nine properties that describe
various settings related to a sensor such as sensor type, sensor class, and sensor value, as
well as the sensor values for upper and lower thresholds. Additionally, if an alert rule was
enabled in the system that matched the crossing threshold level, ILOM would automatically
generate an alert message to the alert destination that you have defined.
ILOM receives error telemetry about error events that occur within the major system
components on the host (CPU, memory, and I/O hub) and the environmental subsystem
within the chassis (such as fans, power supplies, and temperature). The components and
conditions are then diagnosed as fault events and captured in the ILOM event log.
The Component Management features in ILOM enable you to monitor the state of various
components that are installed on the server or managed by the Chassis Monitoring Module
(CMM). You can identify the component name and type, enable or disable it, or check the
current fault status.
Management
Host CPUs
Switch
Service
Processor
Sideband Management
You can connect to the servers service processor (SP) by using the out-of-band network
management port (NET MGT). Alternatively, with the ILOM sideband management feature,
you can select one of the servers Gigabit Ethernet ports (NET 0, 1, 2, 3), which are in-band
ports, to send and receive ILOM commands to and from the server SP. In-band ports are also
called sideband ports. Exalogic compute nodes are configured at the time of manufacturing to
use sideband management via the NET0 port.
The advantage of using a sideband management port to manage the servers SP is that one
less cable connection and one less network switch port is needed. In configurations where a
great number of servers is being managed, such as data centers, sideband management can
represent significant savings in hardware and network utilization. One potential disadvantage
is that while the server is being powered on, you may experience a brief interruption in ILOM
network connectivity.
You require at least one static IP address for Service Processor ILOM access. If you
reconfigure these IP addresses during the initial configuration of the Exalogic machine that is
using Oracle OneCommand and its associated tools, you must use those IP addresses to
access ILOM.
http://<mgmt_address>
1
2
3
1
3
2
Management Settings
By using the Configuration tab you can customize ILOM communication on this device. From
the Web Server subtab, enable/disable the ILOM web interface, enable/disable SSL, or
change the default web server ports (80/443). From the Configuration > Network tab, change
the network settings for ILOM, including the IP address, subnet mask, and gateway address.
You can also change the physical port that ILOM binds to. Keep in mind, however, that Oracle
only supports sideband management for Exalogic (using NET0 by default).
Additional settings are available from the Configuration tab to:
Enable/disable the ILOM command-line interface (CLI) and to configure inactivity
timeout
Configure SSL certificates for HTTPS and SSH access to ILOM
Enable and configure SNMP and/or IPMI client access
Role Privileges
ILOM Roles
For backward compatibility, ILOM 3.0 supports ILOM 2.x user accounts such that users with
ILOM 2.x Administrator or Operator privileges are granted ILOM 3.0 roles that match those
privileges. The 2.x Administrator role maps to Admin (a), User Management (u), Console (c),
Reset and Host Control (r), and Read Only (o). The 2.x Operator role maps to Console (c),
Reset and Host Control (r), and Read Only (o).
1
2
Provide a URL or
upload from your
client machine.
Term Definition
MB Motherboard
P Processor
D Memory module
FB Fan board
ILOM Glossary
Below are some examples of ILOM sensors and indicators:
/SYS/MB/P1/SERVICE Host CPU service LED. If the CPU fails, this LED illuminates
amber
/SYS/MB/P1/D1/PRSNT DIMM 1 on host CPU 1 is present.
/SYS/FB/FM1/OK This is a fan OK LED. It illuminates green to indicate that the fan module
is operating properly.
/SYS/DBP/PRSNT Disk backplane is present sensor.
/SYS/DBP/HDD1/SERVICE Hard disk 1 service LED. Illuminates amber when the hard
disk needs servicing.
/SYS/PS1/VINOK The voltage of power supply 1 is OK.
2
View current
LED state.
3
1
2
Alert Concepts
Each server platform is equipped with a number of sensors that measure voltages,
temperatures, and other service-related attributes about the system. ILOM automatically polls
these sensors and posts any events crossing a threshold to an ILOM event log. It also
generates alert messages to one or more customer-specified alert destinations.
ILOM supports alerts in the form of Intelligent Platform Management Interface (IPMI) PET
alerts, Simple Network Management Protocol (SNMP) traps, and email notifications. Alerts
provide advance warning of possible system failures.
In ILOM, you can configure up to 15 alert rules by using the ILOM web interface or CLI. For
each alert rule you configure in ILOM, you must define three or more properties about the
alert, depending on which of the three alert types you choose.
Alert levels act as a filter mechanism to ensure alert recipients only receive the alert
messages that they are most interested in receiving. Each time you define an alert rule in
ILOM, you must specify an alert level. The lowest level alert generates alerts for that level and
for all alert levels above it. ILOM does not support alert-level filtering for SNMP traps.
The Down alert level generates alerts for only upper non-recoverable and lower non-
recoverable events. The Disabled level disables the alert altogether.
1
2
3
4
Configuring Alerts
1. Click the Configuration tab.
2. Click the Alert Management subtab.
3. Under Alerts, select one of the empty table rows and click Edit.
4. Provide an alert Level and Type. For each IPMI PET alert you configure in ILOM, you
must specify an IP address. For each SNMP trap alert, you must specify an IP address,
a port number, the version of SNMP to use, and an SNMP user/community name. For
each email alert, you specify an email address, the name of the sender, and, optionally,
any advanced filtering criteria.
5. When you finish, you can test the alert by using the Test Rule button.
$ ssh root@exl3cn01-ilom
...
-> show /SYS/T_AMB
Properties:
type = Temperature
ipmi_name = T_AMB
class = Threshold Sensor
Answer: a, b, d
Answer: d