How Microsoft does

“Software
Engineering”

Myung Ho Kim
National Technology Officer (NTO)
Microsoft Korea
Overview

 ProductGroup Organization
 Engineering Software Products
 Hard Decisions and Wisdom
Product Group Organization

Group

Division
Feature-set
Product Unit User Ed &
Product Unit
Marketing Localization

Program
Management Test Dev
Architecture
Implementation
Build BVT QFE

Testing
Quality assurance
A ‘Typical’ Product Group
 25% Developers
 45% Testers
 10% Program Management
 10% User Education / Localization
 7% Marketing
 3% Overhead
The Developer Division
(DevDev)
 Has about 2,200 people
 Visual Studio / .NET Framework: 30+ Product Units

.NET CF .NET Client .NET SDK

CLR ClickOnce DDUE
DDCPX MSDN NCL
PSDK Setup DataWorks
Team System TS: Architect TS: Dev/Test
(Overall)
TS: Team Foundation Visual Studio for Office VB

VC# VC++ VJ#

VS Core VS Data VSDE
WebData XML WebPT WMI
XES xM Localization
DirectoryServices
VS 2005 & .NET Framework 2.0
 Over 8 million lines of code
 Over 15,000 files
 Over 150 binaries
Typical Project Cycle

Vision Feature RTM

statement complete
Releas
Plannin M1 M2 e
g Testin
g

Technology Beta 1 Beta 2 RC1

Preview
Planning Phase
 Marketing gathered requirements.
 Senior leadership provided an initial vision.
 A cross discipline team created a vision
statement and elaborated on product
themes.
 The same team created an overall
schedule.
 Product Units began a detailed planning of
what features would get into the release.
Scheduling *
 Milestone = unit of product cycle progress
– Enable accurate assessment of progress and distance left
 Milestone features are scheduled in priority order
– Enable flexible scheduling to respond to feedback later
 Milestone is done when quality exit criteria is met
– Forcing function to ensure team doesn’t go fast and loose
– Ensure very stable, usable, and complete product at exits
Key Feature Milestone Events

Milestone Event Definition

Spec Complete Date design specs for milestone features written &
reviewed
Feature Coding Typically 8-9 weeks in length for feature milestones

Code Complete (CC) Date all features scheduled for the milestone are finished

Test Plan Complete Test plans for all milestone features written and reviewed

Test Case Code Date all test cases in the Test Plan are completed
Complete
ZBB Test Pass (ZBB TP) All existing functional tests are run on current builds

Zero Bug Bounce (ZBB) # of milestone bugs older than 48 hours = 0

Zero Resolved Bugs # of resolved milestone bugs to be verified before closing

(ZRB) =0
Test Sign-Off Final verification and media sign-off on the milestone
build
Design Specs
 The feature-set is driven by Program
Managers
– Own the customer experience and making the
right thing happen
– Drive the product unit team during the design
process
– Own writing a design specification for each
feature
Coding
 Developers use a variety of tools for coding.
– Visual Studio, Emacs (huh!), vi (huh!!), Source Insight…
– No tool-specific files or code-injection allowed to be checked-in
 Maintain single source tree for entire product line.
– Can type “build -c” from root to build from scratch
– Can type “build -c” in any sub-directory of source tree
 Each team maintains a private branch off main tree.
– Use internal source control system optimized for
branching/merging
– Minimizes check-ins into main tree and avoids breaks
Coding
 C++ & C# are the 2 common languages used.
 The goal is efficient, clean, and maintainable
code.
– Code we ship lives at least 10 years guaranteed
support
 Documentation and resource strings are
stored externally from source code.
– Avoid documentation writers colliding with
developers
– Localization work handled centrally in Ireland &
Japan
– .NET Fx is localized into 34 languages, VS 2005
into 8
Coding
 All code check-ins must be reviewed by
another developer on the team prior to
submitting any changes to the source tree.
 Dev is responsible for “check-in suites” to
exercise implemented features.
 All product check-in suites must run and
pass 100% before any check-in can be
submitted into source tree (2-3 hour
process)
 “Check-in mail” is sent to the team
summarizing changes made, bugs fixed, and
files modified.
Producing Daily Builds *
 A new “build” is built and released every day.
– Forces engineering discipline, critical in the end-game
– Build number indicates build date (40730: Jul. 30th )
 Central build lab produces daily builds for entire
division
– Staffed 24 hours a day throughout the week
 Build process:
– Build team syncs source code tree (~mid-night)
– Kick off end-to-end build (~4:00am)
– Build complete (~1:00pm)
– Perform BVT run to verify build works (~4:00pm)
– Pick up hot-fixes & re-build for BVT failures
– Repeat hot-fix/BVT cycle until the build passes clean
Testing
 Test team is staffed by Devs
– Designs test plans, writes tests, and builds infrastructure
 Target goal with tests is to reach >70% product code
coverage by end of the product cycle
– We measure this statistic throughout the product
– In Visual Studio 2005 - 62% (native) and 70% (managed)
 The Visual Studio 2005 test status:
– 10,339,207 functional test cases
– ~9,000 servers in test lab to run them in automated way
– A full test pass takes 21 days
 Test management system manages test plans/runs
“Mad-dog” Test Automation
System
Bug Tracking
 Bugs & work-items are tracked within an
internal bug-system “Product Studio”.
– Enables rich reporting and change history
tracking
 Bugs triaged by feature leads and assigned
priority
– Bugs fixed in priority order of range from 0 to 4
 Daily status mails sent to entire division
tracking bug status
– After the final feature-milestone is completed,
life of the product team revolves around driving
it to zero.
Ship Mode: Coasting In
 Large projects “coast in” r.t. ending suddenly.
– You can’t dock a large ship abruptly.
 Key set of steps along the “coasting path”
– Lock feature-set and stop adding/changing design
– Run a full test pass to find all bugs of the locked
design.
– Push for a zero bug bounce (ZBB) on the locked
design.
– Absorb bug-bounce over few weeks from ZBB.
– Triage all bugs out of the system that are not “must
fix”.
– Enter End-Game mode & start minimizing code
churn.
End-Game Mode
 “War Team” takes over to the End-Game mode.
– Made up of most senior team members, all with ship
experience.
– Guides the ship in into port.
– Meets at least once a day, ≥ twice a day at the very end.
 Over the span of the final weeks before a release
– War team steadily raise the triage bar, and towards the
end only allow “showstopper” bugs to be fixed.
– If product is ready to ship, the number of showstopper
bugs will slowly decline as check-in rates drop.
End-Game Mode
 Shipping is more of an art r.t. a science
– Senior leaders need to feel that the product is right,
have confidence in test coverage, customer
scenarios…
 Once the rate of incoming bugs gets to zero, put
the build in escrow for a few days, and wait &
see.
– If it holds, ship it.
– If not, repeat until we are finally there.
 Milestone ends
 Rinse and repeat (Iterative process!)
– Visual Studio 2005 went through M0 (planning), M1,
M2, Beta 1, M3.a, M3.b, M3.c, Beta 2, RTM
From Bugs to Builds

Component Dev coding & Buddy Code

triage Unit testing testing check-in

Daily
Bug found
or escalated War Team build

Build CDs BVT

available RTW Pkg Dogfood
Customer Connections
 We realized that our customers need to be
part of our development.
– We had to break the pattern that let us think
that “we know what’s best”.
 Four areas of focus:
– Product Feedback Center (Ladybug)
– Self-sustaining Communities (Forums)
– Community Technology Previews (CTP)
– Technical Assistance Program (TAP)
 Refer to site
http://blogs.msdn.com/ddcpxblg
Hard Decisions
 The Golden Rule
– Features + Quality = Time
You can only control two from these three!
 Development focus had (always) been Q &
F.
– Don’t worry too much about the schedule
 Front Loading Quality
– Maintain a very high quality standard
throughout the project to mitigate F/Q/T
tradeoffs
– No feature work items get checked-in until
they are completely passing tests; there is
always next release
More Wisdom of the Ages
 Products ship with bugs...
– The last bug is found when the last customer stops using
your product
 People will always screw up a good project
– Don’t add them unless you really have to
– “Mythical man month”
 The world is not so static from specs thru RTM
– Resources, priorities, markets, features… will change
– Focus on what ultimately matters
• Quality
• Compelling to customers
• Solving business problems
A Decade of Advances in Software
Construction
and Microsoft’s Scorecard
1. Design has been raised a level
2. Daily build and Smoke test
3. Standard libraries
4. Visual programming innovation
5. Nice community of programmers
6. The Web, for research
7. Widespread use of incremental development
8. Test-first development
9. Refactoring as a discipline
10. Faster computers
-- Source: Construx, Code Complete 2: Realities of Modern Software Construction, 2004
Software is built by People
 Understand what’s really bad for
morale
– Long hours and high stress
– Unrealistic schedules
– Feeling of low productivity
– Working on poor-quality software
Software is built by People
 Watch out workplace demons!
– Dishonesty
– Sloppiness
– Backbiting
– Intolerance
– Jealousy
– Sexism
– Carelessness
– Inability to speak to power
– Lack of respect for the other
– Blaming
 Please come out from the color of the
night…
Closing Remark
 Factors that impact any successful
project
– Technology

– Process

–People