SVR333

Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon

Mark Russinovich Chief Software Architect Winternals Software
Copyright 2006 Mark Russinovich

Mark Russinovich
Co-founder and chief software architect of Winternals Software (www.winternals.com) Co-author of Windows Internals, 4th edition and Inside Windows 2000, 3rd Edition with David Solomon Author of tools on www.sysinternals.com
Home of Marks blog and forums

Microsoft Most Valuable Professional (MVP) Senior Contributing Editor to Windows IT Pro Magazine Ph.D. in Computer Engineering

David Solomon
President of David Solomon Expert Seminars (www.solsem.com)
Founded in 1992 1982-1992: VMS operating systems development

Teach public and private live classes on Windows Internals and Advanced Troubleshooting Microsoft Most Valuable Professional (MVP) Books
Windows Internals, 4th edition Inside Windows 2000, 3rd edition Inside Windows NT, 2nd edition Windows NT for OpenVMS Professionals

Videos
Windows Internals COMPLETE New! Sysinternals Video Library (see DVD in bag)
4

Outline
Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

Troubleshooting Application Failures

Most applications do a poor job of reporting filerelated or registry-related errors
Permissions problems Missing files Missing or corrupt registry data

Errors manifest in several different ways

Misleading error messages Crashes Silently exiting Hangs

Troubleshooting Application Failures

When in doubt, run Filemon and Regmon!
Filemon monitors file I/O Regmon monitors registry activity

Ideal for troubleshooting a wide variety of application failures Also useful for to understand and tune file system and Registry access
Understand hard drive activity Optimize application installation and configuration

Filemon and Regmon run on Microsoft Windows 95, Windows 98, Windows Me, Windows 2000, Windows XP, Windows Server 2003, x64 64-bit Editions, Windows Vista
7

Using Regmon/Filemon
Two basic techniques:
Go to end of log and look backwards to where problem occurred or is evident and focused on the last things done Compare a good log with a bad log

Often comparing the I/O and Registry activity of a failing process with one that works may point to the problem
Have to first massage log file to remove data that differs run to run
Delete first 3 columns (they are always different: line number, time, process id) Easy to do with Microsoft Office Excel by deleting columns

Then compare with FC (built in tool) or Windiff (Resource Kit)

8

Outline
Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

How Filemon Works

Filemon is based on a file system filter driver
Extracts the driver to \Windows\System32\Drivers Installs the driver Deletes the driver file

Requires Debug Programs user right

First run requires the Load Driver user right
Application User Mode Kernel Mode Filemon Driver

Filemon GUI

File System Driver

10

Using Filemon
# - operation number Process: image name + process id Request: internal I/O request code Result: return code from I/O operation Other: flags passed on I/O request

11

Controlling Filemon
Start/stop logging (Control/E) Clear display (Control/X) Open Microsoft Internet Explorer window to folder containing file:
Double click on a line does this

Find finds text within window Save to log file Advanced mode Network option

12

What Filemon Monitors

By default Filemon traces all file I/O to:
Local non-removable media Network shares

It saves all output for display

Can exhaust virtual memory in long runs You can limit captured data with history depth

You can limit what is monitored:

What volumes to watch in Volumes menu What paths and processes to watch in Filter dialog What operations to watch in Filter dialog (reads, writes, successes and errors)
13

Filemon Filtering and Highlighting

Include and exclude filters are substring matches against the process and path columns
Exclude overrides include filter

Be careful that you dont exclude potentially useful data

Capture everything and save the log Then apply filters (you can always reload the log)

Highlight matches all columns

14

Understanding Disk Activity

Use Filemon to see why youre hard disk is crunching
Process performance counters show I/O activity, but not to where System performance counters show which disks are being hit, but not which files or which process Filemon pinpoints which file(s) are being accessed, by whom, and how frequently

You can also use Filemon on a server to determine which file(s) were being accessed most frequently
Import into Excel and make a pie chart by file name or operation type Move heavy-access files to a different disk on a different controller

15

Polling and File Change Notification

Many applications respond to file and directory changes
A poorly written application will poll for changes A well-written application will request notification by the system of changes

Polling for changes causes performance degradation

Context switches including TLB flush Cache invalidation Physical memory usage CPU usage

Alternative: file change notification When you run Filemon on an idle system you should only see bursty system background activity
Polling is visible as periodic accesses to the same files and directories File change notification is visible as directory queries that have no result
16

Demo: Change Notify

Explorer posts change notify to know when directory contents change for open Internet Explorer windows
Open Internet Explorer window to a folder Create or delete a file

17

Demo: Understanding Notepads File Save

1. Run Filemon

2. Set filter to only include Notepad.exe

3. Run Notepad 4. Type some text

5. Save file as test.txt

6. Go back to Filemon 7. Stop logging

8. Set highlight to test.txt

9. Find line representing creation of new file Hint: look for create operation
18

Basic vs. Advanced Mode

Basic mode massages output to be sysadminfriendly and target common troubleshooting Things you dont see in Basic mode:
Raw I/O request names Various internal file system operations Activity in the System process Page file I/O Filemon file system activity

19

Example: Word Crash

While typing in the document Microsoft Office Word XP would intermittently close without any error message To troubleshoot ran Filemon on users system
Set the history depth to 10,000 Asked user to send Filemon log when Word exited

20

Solution: Word Crash

Working backwards, the first strange or unexplainable behavior are the constant reads past end of file to MSSP3ES.LEX

User looked up what .LEX file was

Related to Word proofing tools Uninstalled and reinstalled proofing tools and problem went away

21

Example: Build Fails

While building a program using nmake on a command line link reported an error:
error writing to program database, check for insufficient disk space, invalid path, or insufficient privileges

22

Solution: Build Fails

Saw sharing violation in Filemon:

Performed a handle search for the file in Process Explorer

Saw Windbg had it opened from an earlier debug session even though debug session was closed Closed Windbg

23

Example: Useless Excel Error

Message
Excel reports an error Unable to read file" when starting

24

Solution: Excel Error Message

Filemon trace shows Excel reading file in XLStart folder
All Microsoft Office apps autoload files in their start folders

Should have reported:

Name and location of file Reason why it didnt like it

25

DLL Problems
Process Explorer may solve a DLL versioning issue, but may not if:
A DLL is missing The order of DLL loads is relevant

So, use Filemon!

Look at the last DLL opened before the application died Compare the startup of a working with a failing application
Missing or inaccessible DLLs often not reported correctly Look for NOTFOUND or ACCESS DENIED May be opening wrong versions due to wrong versions being in folders in PATH
26

Example: Word Dies

Word starts and a few seconds later gets a Dr. Watson (access violation)
Customer tried re-installing Microsoft Office still failed

Solution:
Ran Filemon, looked at last DLL loaded before Dr. Watson It was a printer DLL Uninstalled printer problem went away

27

Problem: Perfmon Hangs

Perfmon hung when starting IIS performance counter DLL was last thing Perfmon referenced

28

Solution: Perfmon Hangs

Services snapin showed IIS was hung starting

Investigation revealed an IIS configuration problem

29

Example: Help Fails

The Help command in an application failed on Windows 95, but worked fine on Windows 98/Windows Me/Windows NT4/Windows 2000/Windows XP
Failed with meaningless error message

Ran Filemon on failing system and working system

Reduced log to file opens Compared logs

30

Solution: Help Fails

At the point logs diverged, looked backwards to last common thing done
An OLE system DLL was loaded Noticed this OLE DLL was loaded from a directory in the users PATH on Windows 95, but from \Windows\System on other versions

Conclusion:
DLL loaded on Windows 95 system was not for Windows 95 Got proper version for Windows 95, problem went away

31

Example: Access Hangs

Problem: Access would hang when trying to import an Excel file
Worked fine on other users workstations

Traced startup of Access on failing and working systems

32

Solution: Access Hangs

Compared logs and looked for first unexplainable difference
First unexplainable difference was that Accwiz.dll was being loaded from two different directories Failing system was loading an old Access DLL from \windows\system32 due to having installed older Access previously

Solution: Removed DLL in \windows\system32 and problem went away

33

Example: Pinnacle Studio Hangs

User had a hang when launching Pinnacle Studio Filemon showed accesses to CyberPatrol's DLL, an Internet filtering tool

Cyberpatrol monitors processes by loading a DLL into them

Uninstalling CyberPatrol fixed the problem
34

Example: Misleading AOL Error

AOL worked in one users account, but failed with this on another account on the same system:

User reinstalled AOL, but problem persisted

35

Example: Misleading AOL Error

User did not have admin rights to AOL directory

This version of AOL was not limited-user account friendly

36

Example: Microsoft Office Outlook Application Error

For example, an Outlook application failed with this error:

Ran Filemon and found it was getting Access Denied

Someone had misread a request to remove EDIT rights and removed all rights
37

Example: Microsoft Software Installer Misleading Error

User received this message trying to install something:

Filemon showed the real reason:

38

Demo: Permission Problems and Misleading Error Messages

1. In Explorer, create a folder c:\noaccess
2. Remove all rights to the folder 3. Run Notepad & type some text

4. Run Filemon set filter to Notepad.exe

5. In Notepad, File->Save As to c:\noaccess\test.txt Note error reported

6. Look at Filemon trace and find Access Denied

39

Outline
Introduction Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

40

Configuration Problems
Missing, corrupted or overly-secure Registry settings often lead to application crashes and errors Some applications dont completely remove registry data at uninstall Regmon may yield the answer

41

How Regmon Works

Regmon uses a driver to intercept Registry operations Up until now Regmon has relied on system call hooking to intercept Registry accesses
Hooking isnt supported by the kernel As of Windows XP the system call table is write-protected by default if a system has <256 MB, requiring a trick

Application

Regmon GUI

Regmon Driver Registry Subsystem

Windows Server 2003 introduces a Registry callback mechanism

Driver can see and modify Registry behavior Latest version of Regmon comes with two drivers: one for Windows Server 2003 and one for previous versions Defined in a DDK header file and used by antivirus products

42

Regmon
UI is similar to Filemon Request: OpenKey, CreateKey, SetValue, QueryValue, CloseKey Path
HKCU=HKEY_CURRENT_USER (per-user settings) HKLM=HKEY_LOCAL_MACHINE (system wide settings)

Result return code from Registry operation Other extended information or results

43

Polling and Registry Change Notification

Many applications want to respond to Registry changes
Polling the Registry is just as bad for performance Applications can request to be notified of changes

Like with Filemon, Regmon should be idle on an idle system

44

Registry Troubleshooting
If you suspect registry data is causing problems, rename the key and rerun the application
Most applications recreate user settings when run In this way, the data wont be seen by the application Can always rename the key back

Use Regmon to discover application settings location

45

Demo: Finding Notepads Settings

1. Run Notepad

2. Change Font
3. Run Regmon and filter to Notepad.exe 4. Exit Notepad

5. In Regmon log, find location of user-specific

Notepad settings 6. Double click on a line to jump to Regedit 7. Delete top level Notepad user settings key 8. Re-run Notepad and confirm font resets to default setting
46

Example: Missing Word Toolbar

Problem:
User somehow disabled all toolbars and menus in Word No way to open files, change settings, etc.

Solution:
With Regmon, captured startup of Word Found location of user-specific settings for Word Deleted this Registry key Re-ran Word, which recreated user settings from scratch

47

Example: Misleading Internet Explorer Error Message

Internet Explorer failed to start with this error:

First, looked on system for ICFGNT.DLL

Not there Not on other systems in the network, either

48

Solution: Misleading Internet Explorer Error Message

Captured Regmon trace and looked backwards from end of Regmon log Saw query of Completed value in Internet Connection Wizard key
Value read was 0 Value was 1 on other systems

Solution: Set value to 1 and problem went away

49

Example: Internet Explorer Hangs

Internet Explorer hung when started unless user manually dialed ISP Captured a Regmon trace and looked backwards from point Internet Explorer was hung
Found references to ATT under a RAS PhoneBook key

Solution: renamed ATT key and problem went away Conclusion: previous ISPs dialer had left junk behind

50

Example: Misleading Microsoft Visual

Basic for Applications (VBA) Error
User got this error installing an application:

Regmon showed permissions problem

Solution: Edited permissions

51

Missing Settings
Sometimes queries to what is not there is more interesting than what is there
Identify missing Registry keys
Search for status NOTFOUND

May reveal hidden capabilities

52

Example: Hidden Capabilities

User tried installing Compaq utility on nonCompaq hardware

Regmon showed hidden key to override:

Solution: created key and install succeeded

53

Outline
Troubleshooting with Filemon Troubleshooting with Regmon Using Filemon and Regmon Together

54

Filemon and Regmon

Many times its not clear whether a problem is Registry or file related
And sometimes problems involve both Registry and file configuration problems

Always run Filemon and Regmon when troubleshooting

55

Example: Internet Explorer Hangs

Internet Explorer started hanging on certain folders
Hangs were up to a minute Internet Explorer would work normally for a minute and then hang again

56

Solution: Internet Explorer Hangs

Ran Filemon and saw network path error
Contained references to decommissioned computer

Regmon showed icon lookup configured for missing computer Fix: Delete Paint Shop Pro (PSP) browse files and all PSP file associations
57

Running Filemon/Regmon Before Logon

Sometimes need to capture I/O or registry activity during boot, the logon or logoff process
Problem: when you logoff all your processes are terminated

Solutions:
Run Filemon/Regmon in a different logon session
psexec s i d

Run Filemon/Regmon from a service

Use Srvany (Resource Kit)

Use Regmons log boot option

58

Resources
Technical Chats and Webcasts
http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp

Microsoft Learning and Certification

http://www.microsoft.com/learning/default.mspx

MSDN & TechNet

http://microsoft.com/msdn http://microsoft.com/technet

Virtual Labs
http://www.microsoft.com/technet/traincert/virtuallab/rms.mspx

Newsgroups
http://communities2.microsoft.com/ communities/newsgroups/en-us/default.aspx

Technical Community Sites

http://www.microsoft.com/communities/default.mspx

User Groups
http://www.microsoft.com/communities/usergroups/default.mspx
59

Fill out a session evaluation on CommNet and

Win an XBOX 360!

60

 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.