BMC Configuration Management (Marimba) Best Practices and Troubleshooting

Andy Santosa Senior Technical Support Analyst

9/3/2006

Agenda

CM Infrastructure CM Inventory CM Subscription CM Software Distribution (App. Packager and Content Replicator) CM Deployment Manager

9/3/2006

2006 BMC Software

CM Infrastructure

Whats New in Tuner 7.0? Whats New in Transmitter 7.0?

9/3/2006

2006 BMC Software

Whats New in Tuner 7.0?

Status Report
Returns general information (for example: tuner version, release date, VM version, heap-size, arguments, and operating system information) and checks if the tuner is running
http://hostname:7717/workspace?status

Debug Report

Returns specific information and checks for problems with the tuner (for example: tuner properties, thread dump information, license information, RPC configuration and connections)
http://hostname:7717/workspace?debug&<option>=t includes the option&<option>=f excludes the option http://hostname:7717/?debug&config=t http://hostname:7717/?debug&threaddump=t

Log Report

Returns the tuner and channel history logs

http://hostname:7717/workspace?log http://hostname:7717/workspace/http://hostname:5282/Marimba/Current/SubscriptionService?log

9/3/2006

2006 BMC Software

Status Report

9/3/2006

2006 BMC Software

Whats New in Tuner 7.0? (Contd)

Thread Dump
You can get thread dump information remotely from both client endpoints and servers.
runchannel http://hostname:5282/Version7/TunerAdministrator -getthreaddump -tuner hostname:7717 -username admin -password "" -output C:\Temp\threaddump.txt

If you want to generate thread dumps remotely, the Java VM must be launched with the Xdebug Option. On servers (transmitters, proxies, etc.), profiles do not include the -Xdebug option for performance reasons.

Minimal Mode
Minimal mode is now a separate process known as minituner.exe and is no longer part of the tuner. This makes minimal mode more robust and the tuner more reliable.

9/3/2006

2006 BMC Software

Whats New in Tuner 7.0? (Contd)

Improved Robustness and Stability IPC service switched from DDE to named pipe.
Comparing to DDE, named pipe provides a more efficient and generic bidirectional communication mechanism. It does not require applications to be GUI-based, and data exchanged can be in arbitrary format.

Receipt Service BMC Remedy Change Management could initiate a change request/task on a Marimba Tuner endpoint through Deployment Manager and Policy Manager. Tuner is to provide a receipt service so as to leave a meaningful audit trail for the work performed on an endpoint. Receipts are stored in the tuner workspace.

9/3/2006

2006 BMC Software

Whats New in Transmitter 7.0?

More robust Transmitter storage Detection of ungraceful Transmitter shutdowns Tx Verify improvements New debug flags: Tuner TUNER/RECEIPT Transmitter TX/GC TX/DIFF TX/DSL TX/HTTP

9/3/2006

2006 BMC Software

Infrastructure - Recommendations
Separate CMS, Transmitters and Windows Patch Source Java Launch Arguments:
marimba.launch.javaArgs=-Xms128m -Xmx512m -XX:PermSize=32m XX:MaxPermSize=128m

Using Profile Property

If the tuner profile property doesnt exist -> ONLY Kernel Upgrade If the tuner profile property exists, but Transmitter doesnt have the profile segment > No Kernel Upgrade If the tuner profile property exists, but Transmitter has the profile segment -> Kernel upgrade + Profile applied

Health Check Tools

Replication Monitor Tx Verify Transmitter Guardian

9/3/2006

2006 BMC Software

Troubleshooting - Scheduler
Update schedule vs. start schedule
Subscription/Policy Service -> update.schedule Patch Service -> update.schedule Inventory/Scanner Service -> start.schedule Infrastructure Service -> start.schedule

Missed update/start schedule while the machine was off the network/turned off
Example: Subscription update schedule at 2 AM Inventory start schedule at 3 AM Marimba schedule start delay=10000

Schedule dumper to help debugging scheduler issue

10

9/3/2006

2006 BMC Software

CM Inventory

Whats New in Inventory 7.0? Performance Debugging

11

9/3/2006

2006 BMC Software

Whats New in Inventory 7.0?

In 6.x, compliance will ALWAYS re-send all data (even when no data changes between Using AMT Data As mac_id New Diffing Logic
scans). This results in an expensive delete of all compliance data followed by a re-insert of all data. This was fixed in 7.0. Intel AMT data is also now part of Intel inventory scans. Identify if machine is AMT-enabled and fetch persistent Tuner ID from AMT chip. Maintain a full report cache on the endpoint. Diff against the current inventory data in the DB if there is a checksum mismatch. Purge full report cache based on size and/or time interval.

Double-Byte Storage Mirror Forwarding

Mirrors on WAN can forward Inventory reports to LAN Mirrors/Master Reduce the chances of database blocking or deadlocks Scan and report task receipts
9/3/2006 2006 BMC Software

Task Receipts (CCM)

12

Whats New in Inventory 7.0? (Contd)

Old 6.X Diff Over-Write That Resulted in Full Scan Report

[15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)

1
Diff 1 Inventory Service Plugin

DB Diff 1 Disk Queue: Diff 2 DB Diff 1 Inventory Service Plugin

2
Disk Queue: Diff 2 Diff 3

Diff 2

` Endpoint

3
Diff 3 Inventory Service Plugin Diff Checksum Mismatch: Diff Scan Old Checksum is for Diff 2, Plugin Local Checksum Cache has Diff 1. Request Full Scan report from Endpoint

DB Diff 1

4 5
Endpoint sends Full Scan Report Full Scan Report

Disk Queue:

DB Delete All Insert All Inventory Service Plugin

13

9/3/2006

2006 BMC Software

Whats New in Inventory 7.0? (Contd)

7.0 Special Diff Scan Report (Instead of 6.x Full Report)
[15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)

1
Diff 1 Inventory Service Plugin

DB Diff 1 Disk Queue: Diff 2 DB Diff 1 Inventory Service Plugin

2
Disk Queue: Diff 2 Diff 3

Diff 2

` Endpoint

3
Diff 3 Inventory Service Plugin Diff Checksum Mismatch: Diff Scan Old Checksum is for Diff 2, Plugin Local Checksum Cache has Diff 1. Request Special Diff report from Endpoint

DB Diff 1

4
Endpoint generates new special diff scan report between the full report at Diff 1 state and the full report at the Diff 3 state and resends back to the plugin

Disk Queue:

5
Inventory Service Plugin Diff 3'

DB Diff 3'

14

9/3/2006

2006 BMC Software

Performance
SQL Server Report Center queries are basically unusable while inventory
insertions are running in the background.

Highly recommend enabling dirty reads if inventory insertions need to be run

while Report Center queries take place. Reason: Inserts/Updates/Deletes issue an exclusive lock Select statements issue a shared lock A shared lock cannot be issued when an exclusive lock is in place and an exclusive lock cannot be issued when a shared lock is in place You can change the behavior of locking via a lock hint: - Nolock - Enables dirty reads where a Select does NOT issue a shared lock and Select does NOT honor exclusive locks - 6.x Only Report Center interactive queries can use dirty reads (enabled through property) - 7.0 Report Center interactive queries, email reports, collections, and processing queries that another application requests can use dirty reads (enabled through UI)
9/3/2006 2006 BMC Software

15

Performance (Contd)
Inventory Plugin Tuning
Set the scan schedule as far apart as possible (24 hours, if possible). Use the scheduling vary option to reduce the load all insert at the same time load (due to time syncd machines). 3 to 5 inventory plugin Oracle database connections appears to be optimal. More connections just result in more resources and waiting instead of increasing throughput. Disk I/O is low. CPU utilization is low. Memory is low. Java GC tuning is lower priority since currently GCs / heap size growth does not appear to be causing issues.

16

9/3/2006

2006 BMC Software

Debugging Customer Performance Issues

Guide to Debugging Customer Performance Issues
Get inventory plugin logs from all masters/mirrors/repeaters. Gather a few sample Endpoint Inventory Service channel workspaces. Oracle redo logs and/or archive logs (if archive log mode is enabled).
If gathering archive logs, also request character sets, dictionary file, version/platform Oracle is running on.

Tuner/History logs where inventory plugins running. System Architecture Scan schedule, # endpoints, # repeaters, # mirrors, load balancer, which plugins insert directly into database, etc. Gather any queued-up scan reports in the inventory plugin disk queue. Find out if the scan schedule is too aggressive and check if it is possible to reduce the scan schedules. In more complex environments, see if client IP Load Balancer persistence might help reducing checksum mismatches.

17

9/3/2006

2006 BMC Software

CM Subscription

Whats New in Subscription 7.0?

18

9/3/2006

2006 BMC Software

Whats New in Subscription 7.0?

Install Priority
You can now set install priority values by typing numeric values per package on the Edit Policy page. In M6, you can only scroll arrow up and down to change the install priority.

User-Based/Machine-Based Policies
Policy will be assigned based on either user-based, machine-based, or both. In M6, the only option was both. Property: marimba.subscriptionplugin.resolvetype Value: user or machine, otherwise both.

Directory Service Hierarchy Structure

Policy Manager now uses an updated LDAP container structure. Enables you to more effectively extend the schema of the Active Directory infrastructure, and store and secure policies.

19

9/3/2006

2006 BMC Software

Whats New in Subscription 7.0? (Contd)

Supports install order Supports staging new url before doing an autoupdatefrom Will not updatefrom if marimba.subscription.nodelete=true

Will revert url if channel fails to install

20

9/3/2006

2006 BMC Software

Compliance Architectural Overview

Inventory Plug-In Implements differencing for compliance. It only inserts compliance data that has been modified from the previous inventory scan on the endpoint. Subscription Service Invoked as a custom scanner by Inventory Service to calculate compliance on the endpoint. Query-Based Compliance Two types of queries in compliance engine: 1. Inventory-only query; query based solely on inventory scan data 2. Latest-data query; query based on inventory, LDAP-synced policy and Transmitter

21

9/3/2006

2006 BMC Software

Compliance (Contd)

Scheduling LDAP Sync The LDAP synchronization process is new in this release. LDAP synchronization optimizes the accuracy of compliance queries and reports by updating the configuration database with the latest available policy data. To optimize policy management workflow, schedule the following processes in sequence: Update policy service Perform inventory scans Schedule LDAP Sync Typically, you schedule the processes on a daily basis You can schedule LDAP Sync using the LDAP-to-Database Synchronization Service link on the Data Source tab in CMS System Settings Compliance Reporting In addition to viewing policy compliance information based on targets, you can now view compliance based on a specified package.

22

9/3/2006

2006 BMC Software

New Features: Compliance (Contd)

23

9/3/2006

2006 BMC Software

CM Application Packager/Content Replicator

Whats New in Application Packager 7.0? Whats New in Content Replicator 7.0?

24

9/3/2006

2006 BMC Software

Whats New in Application Packager 7.0?

Support for MSI 3.0
No need for User Elevation when installing MSI packages Delaying the download of MSI package until pre-scripts are run Repair of MSI now shows progress bar Return code mapping App-friendly name DSL support

25

9/3/2006

2006 BMC Software

Whats New in Content Replicator 7.0?

Packaging and Installation of Content From
Existing 1 source to 1 target folder

N source to N target folders Installing Unix Packages Using XML Command file to specify packaging commands {5.1 screen shot} Self-Installing Channels New package command that will facilitate installation of self-installing channels {5.2 Screen shot with syntax} Channel Signing (Credit Suisse)

26

9/3/2006

2006 BMC Software

Whats New in Content Replicator 7.0? (Contd)

No More Auxiliary Channels; Now Segments
(Segments also, only for n source - n target use case)

27

9/3/2006

2006 BMC Software

CM Deployment Manager

Whats New in DM 7.0?

28

9/3/2006

2006 BMC Software

Whats New in DM 7.0?

Main Features (Architecture Changes)

Database-backed workspace Improved DS->DM communication Improved handling of log data Report Center integration

29

9/3/2006

2006 BMC Software

Previous Workspace Architecture

File System-Based
Object-tree is stored on disk in File System hierarchy Each object consists of a directory with several files Entire object-tree is loaded up and kept in memory at startup

Problems
Keeping all nodes in memory doesnt scale
Start-up time grows linear to workspace size Workspace size is restricted by available memory

Hard to:
Query data Share data with other applications Maintain referential integrity Guarantee safety of the data

30

9/3/2006

2006 BMC Software

New Workspace Architecture

Database-Backed
Object-tree is stored in a database under DBTree Simple Similar to what DM uses today (PTree). Easy to learn and replace. Yet powerful Lazy loading: In-memory caching and batching of object-updates, etc. An object is a DBTree dir-node All common files are merged into a set of node-properties. Other data stored as child-nodes. Exception: Log data is stored in the logging schema.

31

9/3/2006

2006 BMC Software

DB-Backed Workspace
How Does This Solve The Problems?
Workspace-size scalability Tree no longer needs to be traversed at start up; start up time will be minimal Number of objects no longer limited by amount of memory Querying the data Logging data is stored in logging schema; querying will be easy Object data is stored in DBTree-schema; querying is possible but somewhat complicated Sharing data All data is in the database Referential integrity Yes, we can guarantee it. Safety of data Increases; databases are built to be safe datastores.

32

9/3/2006

2006 BMC Software

DS->DM Communication
Problems
DS communicates too frequently Protocol is inefficient and simplistic DS queue logs in memory

Solutions
Tighter control of DS communication
Intelligent flushing of queue

New, efficient protocol

Binary Support for compression Support for new client commands (re-try later, etc.)

Better use of HTTP

Be efficient about keep-alives (partly done in 603S releases)

33

9/3/2006

2006 BMC Software

Handling of Log Data

Problems
DM does on-the-fly data processing (CPU-intensive) Incoming data is immediately processed UI becomes unusable when many clients upload

Solutions
Incoming data is placed in disk-based queue Limited set of processor threads read off of queue We get in control of CPU resources used for log-processing vs. UI May introduce latency in respect to reporting under high load This should be acceptable since the alternative is an unusable UI

34

9/3/2006

2006 BMC Software

Whats New in Deployment Reports?

Leverage The Fact That Data Is In The DB
Use Report Center for building reports Separate schema for reporting Near real-time reports (for deployment status) After-the-fact reports Reports to be available in DM UI RC query-tree is accessed remotely Report can be selected from DM UI

35

9/3/2006

2006 BMC Software

Q&A

36

9/3/2006

2006 BMC Software