Professional Documents
Culture Documents
The Definitive Guide to SharePoint Logging and Auditing A Technical Support and User Guide for SharePoint Portal Server and Windows SharePoint Services
2006 David M. Sterling All Rights Reserved
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Table of Contents Event Logging in SharePoint ...........................................................................................3 SharePoint Log Timing............................................................................................3 Enabling SharePoint Logging ..................................................................................4 Parsing SharePoint Logs..........................................................................................4 IIS Log information and SharePoint Logs ................................................................5 SharePoint Transactions ..............................................................................................6 Noise Transactions in SPS Logs ...........................................................................6 Document Transactions ...........................................................................................6 List Transactions .....................................................................................................6 IIS Logging .................................................................................................................7 Logging Elements (W3C extended format) ..............................................................7 Setting up Logging ..................................................................................................7 Logging with ODBC..............................................................................................10 Performance Considerations ..................................................................................13 Understanding IIS Logs for SharePoint......................................................................15 Noise Transactions .............................................................................................15 Detecting when a User has selected a New Document............................................15 Opening a file in READ ONLY mode ...................................................................17 Opening a File for Editing .....................................................................................17 Uploading a File ....................................................................................................19 List Operations ......................................................................................................19 Cross Referencing Logs.........................................................................................19 Calculating SharePoint Statistics ...................................................................................20 Determining number of Downloads a user has Performed..........................................20 Using IIS Logs ......................................................................................................20 Using SharePoint Logs ..........................................................................................20 Calculating Storage Usage.........................................................................................21 Calculating Document Library Storage ..................................................................21 Calculating Attachments Storage ...........................................................................22 Calculating Usage For a Site..................................................................................22 Counting the Number of Immediate Alerts Active in the Portal..............................22 Counting the Number of Scheduled Alerts Active in the Portal..............................22 Obtaining Sites a User Belongs To ........................................................................23 Obtaining Site Members ........................................................................................23 Obtaining User Profile Information........................................................................23 Appendix A...................................................................................................................24 Source listing for SharePoint Log Parser....................................................................25 SPSLogParser XML Settings File..............................................................................25 SPS Log Parser Table Definition ...............................................................................25 SPS Log Parse Table Bulk Insert Command ..............................................................25
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
As mentioned under Performance Considerations, the directory/folder you specify should have adequate rights granted to the IIS_WPG, STS_WPG, SharePoint Services and SharePoint Administration accounts.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Time Stamp Site GUID Site Name Web Name Document Name (of the request, ASPX page, Document, etc.) User Login The Date of the log must be determined by the Folder name. Key elements missing include the previous indicator and the Flags indicating the type of hit. In Appendix A you will find a rewritten version that supports automatically bulk loading or inserting directly into the database. In addition, the output is more useful as shown: [siteGuid] GUID of the site (cross reference for the VirtualServers table in the SPS Configuration Database) [time] Time Stamp of the event (in SPS, the Folder Name contains the Date) [siteUrl] Site URL of the request [web] Web Name of the current sub web [doc] Document requested an ASPX document or actual document name (with full HTTP path) [username] User Login name (Domain\Username) [useragent] User Agent information same as IIS [referrer] Referrer information same as IIS [querystring] Query string passed same as IIS [bitFlags] Type of hit 0 = Regular hit, 1 = Used by Office Front Page, 2 = List Update, 4 = List Operations or 8 = Discussion request made through OSE (Office Server Extensions) Discussion button in IE
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SharePoint Transactions
Noise Transactions in SPS Logs
Like IIS below, you can usually omit any entries that are based on the SharePoint Services or Administration account. In addition, you can eliminate default Search entries by checking the Agent for Search the agent is listed as:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Document Transactions
Transaction Type refers to the Bit Flags set in the log file (0, 1, 2, 4 or 8) When a document is opened for editing of the meta data: A Type 1 transaction with Doc = Document Name A Type 1 transaction with Doc = edtiform.aspx A Type 2 transaction with Doc = Library Name When a document is opened for editing: A Type 1 transaction with Doc = Document Name A Type 1 transaction with Doc = owssvr.dll When a document is saved from MS Office to SharePoint, four type 1 transactions occur: The first to owssrvr.dll The second to owssvr.dll with User Agent set to 'Test for Web Form Existence' The third to owssvr.dll The fourth with the 'doc' column set to the document name being saved When a document is Uploaded to the site: A Type 1 transaction with Doc = Upload.aspx A Type 1 transaction with Doc = Document Name uploaded When a document is Deleted: A Type 4 transaction with Doc = Document Library URL (without the file name)
List Transactions
List transactions are easily determined by the Type of transaction: Type 2 indicates the list was updated Type 4 indicates List Operations (save and delete)
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group Page 6 of 25 February 2006
IIS Logging
To support SharePoint auditing, you will need to incorporate information from IIS Logs. There are two ways in which to process these, either by bulk loading the logs into SQL Server or by integrating IIS ODBC logging to add the data immediately. Note that while the better method is to use Bulk Load due to processing time, the ODBC method is preferred for immediate logging and monitoring. Subsequent analysis can use the bulk load method to get a more complete picture. IIS log files are delimited text files that follow the specification RFC2616, Hypertext Transfer Protocol HTTP/1.1 (http://www.rfc-editor.org/rfc/rfc2626.txt).
Setting up Logging
You must enable IIS logging and setup the fields desired using Internet Information Services Manager. Within IIS, right click on the web site to be monitored and select a Logging Option as shown:
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
The two options of interest are the W3C Extended Log File Format or ODBC Logging. After you have selected the method, you must select Properties to set which fields will be logged.
By default, these fields will be WRONG you must set them to match what you want to log and at the same time, you must also setup the database to hold the items you select. Based on the base list shown here, the basic SQL Script to create a capture table is as follows:
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
NOTE: You may encounter an error in SQL Query Analyzer running this script this is because the record is longer than recommended. If this occurs, you must create the table using SQL Enterprise Manager. You can map these one to one to the fields selected in IIS Logging:
NOTE: the order of the fields will depend on your installation to be sure of the order in which your fields are being saved, you must check the Log file itself and locate the #Fields line:
#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) sc-status sc-win32-status sc-bytes time-taken
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
To load the Log file into SQL Server, you must prepare the Log file for import. The first four lines of the Log file contains # as the starting character (for example, the fields line above). These must be removed or the Bulk Insert will fail. These lines can be edited out manually or you can use the Microsoft Pre-built tool for this located at http://support.microsoft.com/default.aspx?kbid=296093. NOTE: The field SC-Win32-Status data type must be either Float or BigInt to accommodate the status fields. Once the file has been prepared, it must be bulk loaded into SQL Server via the Query Analyzer as:
BULK INSERT [dbo].[IISLoggingFormatAdv] FROM 'C:\mystore\PreppedLog.log' WITH ( FIELDTERMINATOR = ' ', ROWTERMINATOR = '\n' )
WARNING: LOG TIMES USE GMT TIME (i.e. 9am EST = 2pm GMT).
ProcessingTime integer
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Target Parameters
NOTE: You may encounter an error in SQL Query Analyzer running this script this is because the record is longer than recommended. If this occurs, you must create the table using SQL Enterprise Manager. Next, you need to create a data source within Windows - this is done via the Administration tool called Data Sources (accessible via Start > Administrative Tools > Data Sources (ODBC). Define a new System DSN that points to your Database including the Login credentials needed. Once defined, you must tell IIS to log via ODBC. Via the Internet Information Services Manager, right click on the Web Site and from Web Site Tab, select ODBC Logging. You will be prompted to enter the ODBC Data Set Name (DSN) as shown:
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
For example, our DSN is called WebLog, has a table name called IISLoggingFormatAdvForODBC and will use the account SPSAdmin:
Note: the account you use should have either DBO access to the database or must have at least db_ddladmin, db_datareader and db_datawriter privledges. After you have setup ODBC, you must test it to be sure it is operating correctly. If you do not see entries recorded, check the Event Viewer under the System log and look for any IIS errors.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Most common problems: User login is incorrect in the DSN definition in Data Sources User does not have sufficient privileges to write to the DB DSN is setup to point to the MASTER database (change to point to the correct default database where the logs are to be written
Be aware that ODBC logging does NOT PROVIDE you with all of the information needed for tracking SharePoint activities two fields, cs(Agent) and cs-uri-query are needed to see file information. See Cross Referencing Logs.
Performance Considerations
There are two sticking points with logging in both IIS and SharePoint first, by default, IIS will create logs in the default Windows folder this is not a good idea for long term logging. This saps precious OS resources and most importantly, disk space. This is also true of SharePoint which uses the LogFiles directory as well. Because of this, it is highly suggested that you setup a separate disk area to contain log files. This should be big enough to accommodate the large files and most important: all IIS/SPS accounts must have FULL access to the folder. This includes process accounts: IIS_WPG, SPS_WPG and if available STS_WPG, Administrative accounts: Administrators, Administrator and SharePoint Accounts: SPS Services and SPS Administrator. In general, logging will always add overhead to IIS operations and naturally, using offloaded files and bulk importing into SQL Server can be much more efficient than
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
ODBC as it will use less system resources and can be offloaded to perform the work without impacting users and is best if you are logging primarily for reporting. ODBC has the advantage of being instantaneous; if you intend to do near real-time, you must use ODBC. Be aware however, that IIS ODBC is not always a perfect connection; in practice, it is common for IIS to skip recording then resume for unknown reasons. If logging is a critical factor, ODBC should be avoided.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
If User does NOT save the document, this will be followed by:
1) A GET request with the UserName filled for the original page user started on (for example if in Area Auto's Document Library, the URL would be /Auto/Document Library/Forms/AllItems.aspx) 2) A GET request with the exact same time as #1, UserName set as - to the same page from #1 returning a 401 and Win32Status of 0
3) A GET request with the exact same time as #1, UserName set as - to the same page
from #1 returning a 401 and Win32Status of -2146893042
NOTE: THERE MAY ALSO BE A REQUEST for /_layouts/images/headcornerp.gif - this is corner image of the Search Box. If the User SAVES the document, this will be followed by:
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
3) A HEAD request with the UserName filled and Parameters set (i.e. location=<doc lib
location>) to owssvr.dll
NOTE: YOU MUST CROSS REFERENCE TO DETERMINE WHICH FILE WAS OPENED.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
NOTE: YOU MUST CROSS REFERENCE TO DETERMINE WHICH FILE WAS OPENED. If User does NOT save the document, this will be followed by:
1) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll)
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
NOTE: THERE MAY ALSO BE A REQUEST for /_layouts/images/headcornerp.gif - this is corner image of the Search Box.
Uploading a File
No IIS Log entries are created when a file is uploaded this is only recorded in the SharePoint Logs.
List Operations
IIS Log Entries for List Operations is not sufficient to determine the actual process; the SharePoint Logs should be used instead.
If a User opened the file and did NOT make any changes, cs-uri-query will be set to -.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Note: Users MAY OR MAY NOT HAVE A PROFILE; such users are orphaned within SharePoint security until a profile is created (if at all). Examples of this: Local system administrator, domain admins, etc.
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Appendix A
Creating the SharePoint Log Parser
Creating this program requires Visual Studio with C# installed.
Executable Creation
Step 1: Create a new C# Windows Console Application project called SPSLogParser Step 2: Copy the Source Listing below into the default class generated Step 3: Rename the class file to be SPSUsageParser.cs Step 4: Add references to System.xml and System.Data namespaces Step 5: Compile
Database Preparation
NOTE: You can use any database for this, but a separate database (stand alone) is suggested. Step 1: Create a new Database called WebLogs (optional) be sure that the SharePoint Administration and SharePoint Services account both have the same access to this database as they do to the SharePoint database. Step 2: Using the Query Analyzer, connect to the database Step 3: Run the Table Creation script shown in SPS Log Parser Table Definition
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group