Intelligent Streaming

Bill Birney Microsoft Corporation

Updated October 2000

Contents Intelligent Streaming ......................................................................................................................... 1 How Does Intelligent Streaming Work?.......................................................................................... 1 Multiple Bit Rate Encoding ......................................................................................................... 2 Intelligent Bandwidth Control .................................................................................................... 2 Intelligent Image Processing ..................................................................................................... 3 How to Author for Intelligent Streaming ........................................................................................ 3 Setting Up the Encoder Using Profiles ........................................................................................ 3 Viewing the Process................................................................................................................... 4 Monitoring Performance............................................................................................................. 5 Custom Profiles.......................................................................................................................... 6 Summary ....................................................................................................................................... 9

Introduction
Intelligent streaming is a set of features in Microsoft Windows Media Technologies that automatically detects network conditions and adjusts the properties of a video stream to maximize quality. Today's Internet connections are highly variable in terms of actual throughput achieved for any specific connection and range of possible connection speeds. For example, a user with a laptop computer can connect to an Internet service provider (ISP) with a 300-kilobit-per-second (Kbps) DSL connection at home, a 1.5megabit-per-second (Mbps) T1 connection at work, and a 56.6-Kbps modem connection while traveling on business. Furthermore, the actual throughput achieved in each of these scenarios is likely to vary. This is especially important for low-bandwidth modem connections, where the connection speed can vary by 50 percent or more of the maximum, depending on network and ISP congestion.

Because Windows Media Technologies is a connected, end-to-end, client/server system, the server and the client communicate with each other to establish actual network throughput and make a series of adjustments to maximize the quality of the stream. Intelligent streaming offers dramatic benefits to the user. It maximizes use of available bandwidth; in a DSL or LAN environment, users receive content tailored to their connection speed. It greatly improves the user experience; users connected by modem immediately notice the presentation is smoother, less jerky, and of generally higher quality.

How Does Intelligent Streaming Work?

Intelligent streaming can automatically adjust between multiple video bit rates and clean up video streams. The most difficult task of streaming audio and video over a network is maintaining a continuous presentation to the user in a highly changeable environment. Buffering is the biggest problem of streaming digital media. It is caused when Microsoft Windows Media Player, the client, runs out of data and must wait

for more to arrive. The client will always run out of data if the bit rate of the stream exceeds the current available bandwidth.

Unpredictability of bandwidth is taken for granted on the Internet. A user can, for example, originally connect to an ISP at 56 Kbps. Just because the connection speed is fast does not mean the bandwidth supports the bit rate. Actual bandwidth is determined by network conditions, and traffic on the Internet is constantly fluctuating, causing bandwidth to plunge to 18 Kbps one moment, then increase to 40 Kbps the next. If a user attempts to view video being streamed at 50 Kbps, the presentation suffers considerably when bandwidth is squeezed.

To ensure a continuous presentation, you must employ a system that adjusts the bit rate to changes in available bandwidth. Intelligent streaming does this by:

Sending a stream with the appropriate bandwidth when the user first connects. Dynamically and seamlessly adjusting the bit rate as the bandwidth changes.

Multiple Bit Rate Encoding

To take full advantage of intelligent streaming, content must be encoded at multiple bit rates. In multiplebit-rate encoding, up to 10 discrete, user-definable video streams and one audio stream are encoded into a single Windows Media stream. The video streams are encoded from the same content, but each is encoded for a different bit rate. When a multiple-bit-rate Windows Media file (with a .wmv file name extension) or live stream is played on Windows Media Player that is connected to a Windows Media server, only one of the video streams is received: the one that is the most appropriate for current bandwidth conditions. The process of selecting the appropriate stream is completely invisible to the user, and this is what intelligent streaming is all about.

Intelligent Bandwidth Control

There are a number of steps in the intelligent streaming process. Each is a strategy-a way to modify the bit rate so it remains continuous on the client end regardless of the current bandwidth. As bandwidth fluctuates between server and client, the server detects the changes and adopts the best strategy. When bandwidth is at its best, the server employs the first strategy. As conditions worsen, the server checks its list of options one by one until the bit rate is optimized for the current available bandwidth.

Intelligent streaming uses the following strategies: 1. 2. The server and client automatically determine the current available bandwidth, then the server selects and serves the video stream at the appropriate bit rate. During transmission, if the available bandwidth decreases, the server automatically detects the change and switches to a lower-bandwidth stream. If bandwidth improves, the server switches to a higher-bandwidth stream, but never higher than the original bandwidth. If bandwidth can no longer support streaming video, the client and server intelligently degrade image quality.

3.

When a network is extremely congested, intelligent streaming attempts to maintain a continuous audio stream. The server decreases the video frame rate to minimize interruptions caused by buffering. If the bit rate is still too high, the server stops sending video frames. If audio quality starts to degrade, the client intelligently reconstructs portions of the stream to preserve quality.

Intelligent Image Processing

The client intelligently post-processes the video stream to enhance quality even at very low bit rates. Windows Media Technologies include a new intelligent filtering technology, which works in conjunction with the Microsoft Windows Media Video 7 codec in Windows Media Player to smooth blockiness and remove ghosting artifacts, significantly improving the overall appearance of the video.

Blockiness also occurs during the decoding of high-bit-rate streams, but it is not as noticeable. A streaming media codec such as the Windows Media Video 7 codec, encodes a video image by breaking it up into pixels. The lower the bit rate, the fewer the pixels. When too few pixels are used to create an image, they appear as blocks. The client post-processing filter used in intelligent streaming smoothes the edges of the blocks and erases certain other artifacts, such as ringing, so the resulting image is more pleasing to the eye.

How to Author for Intelligent Streaming

To encode with multiple bit rates, simply use one of the multiple-bit-rate profiles provided in Windows Media Encoder 7. For greater control, you can manually select the exact bit rates for each of up to ten encoded streams. Client post-processing and intelligent bit rate optimization are all automatic, on-the-fly features. Best of all, you only need to create and manage a single file to handle multiple bit rates.

To understand the following topics, you should be familiar with the operation of Windows Media Encoder 7. For more information, see the Help documentation that is included with Windows Media Encoder 7. Windows Media Encoder 7 is available for download from the Windows Media Web site .

Setting Up the Encoder Using Profiles

You can tell which of the system profiles (the ones that are installed with Windows Media Encoder) encode multiple bit rates by reading the description of each on the Profile Selection screen of the New Session Wizard, or the Profile tab of the Session Properties dialog box. For example, the description for the Video for dial-up modems or single channel ISDN (28.8 to 56 Kbps) profile reads: "Use this multiple bit rate profile for target audiences with a dial-up modem or single channel ISDN connection (bandwidth is between 28.8 Kbps and 56 Kbps)." Using this profile, the encoder creates several streams that cover users connecting at speeds between 28.8 Kbps and 56 Kbps.

Note that the profile list does not show multiple-bit-rate profiles if you are encoding audio only or you have selected File will stream from a Web server or play directly on a computer on the Output File Distribution screen of the New Session Wizard. Intelligent streaming only applies to video that is streamed from a Windows Media server.

See Windows Media Encoder 7 Help documentation for more information about editing or creating new profiles that encode with multiple bit rates.

Viewing the Process

Before encoding, open a performance monitor on your system. In the Microsoft Windows NT or Windows 2000 operating systems, right-click the taskbar, click Task Manager, and then click the Performance tab. There are similar performance monitor options available for Microsoft Windows 98.

With the multiple-bit-rate profile selected, the encoder configured and ready, performance monitor running, and video and audio streams connected and adjusted, start encoding. If your computer has dual 400-MHz processors, CPU usage is 35 percent to 40 percent. With usage in that range, the encoder has enough processing power to handle rapid increases in frame detail. If you are encoding on a slower computer, however, you are pushing the capability of the processor. Using a single 200-MHz processor, for instance, CPU usage immediately climbs to 100 percent when encoding starts. While this condition is normal when encoding from one file to another, high CPU usage when encoding live usually means frames are being dropped or discarded.

To monitor the encoder as it is working, view statistics on the Monitor panel.

Figure 1. Encoder Monitor

Certain fields in this panel change continuously as encoding proceeds. Under Encoding, the Media Bit Rate can change dramatically as the amount of video detail per second increases. This illustrates how the encoder continuously adjusts parameters to maintain the current bit rate as closely as possible to the bandwidths selected in the profile. The Media Bit Rate shows the combined bit rates of all the streams being encoded at a particular point in time.

A multiple-bit-rate Windows Media file is larger than a single-bit-rate file of the same length because of the extra streams. Likewise, the bandwidth of the connection between the encoder and the server must be larger to handle more streams. In this example, the system profile calls for six multiple-bit-rate video

streams and one audio stream to be encoded, for a total of seven streams. However, only two streams are delivered by the server to a client: the audio stream and one video stream.

When a client attempts to connect to the server to receive the live presentation, the server determines the current bandwidth of the connection. For example, suppose a user connects with a 56-Kbps analog modem, but does so at a time when network traffic is particularly high. When the user connects, the server might determine the actual available bandwidth to be 40 Kbps. At that bandwidth, the 53-Kbps stream is too high, so the server streams the 31-Kbps video stream and the audio stream.

After 10 minutes, network congestion increases, and bandwidth suddenly falls to 32 Kbps. Frame rate begins to suffer, and some packets are lost, but the server reacts immediately by switching to the 19-Kbps stream. The user notices some degradation in image quality, but audio is continuous and disruption of the presentation is minimized. 20 minutes later, bandwidth worsens again, dipping down to 14 Kbps. This is even too low for the lowest video stream, so the Windows Media server stops all video. The user notices the loss of video but is still able to listen to an uninterrupted audio stream, which only requires 8 Kbps of bandwidth. A few minutes later, bandwidth improves a great deal, and the server again is able to send the 31-Kbps video stream.

The negotiation between server and client is handled automatically and seamlessly. There are no manual adjustments necessary on either end. If multiple-bit-rate streams are available to the server, it uses them. The only thing you have to do as producer is make sure the streams are there. If one of the multiple-bitrate profiles is not exactly right for your needs, you can edit it or create a new profile in the Manage Profiles dialog box. You can open this dialog box from the Profile tab of Session Properties or the Profile Selection screen of the New Session Wizard.

Monitoring Performance
In the multiple-bit-rate environment, the single most important concern is CPU speed. While this was certainly a factor in the single stream, low-bit-rate days, it is crucial now. It is recommended that you invest in a computer with a processor speed of 400 MHz or greater and dual processor, if possible. A slower computer (for example, in the 200-MHz category) is suitable if live encoding is limited to only two of the lowest bit rates and file encoding time is not an issue. In a multiple-bit-rate environment, the more CPU speed available to the encoder, the more streams are possible; and the higher the bit rate and frame rate, the bigger the frame sizes and the better the quality.

Monitoring the performance of your CPU and memory resources is a simple way to monitor the quality of your encoding. Poor playback of your live video can often be attributed to an overloaded CPU. When you select an encoding profile, you are, in effect, assigning a list of tasks for your CPU to perform. The more streams, the more frames per second, the larger the image size and the higher the quality you enter, the more tasks per second your CPU must perform. Windows Media Encoder automatically adjusts its task load to the given bandwidth and to the limits of the CPU. For example, if you enter a high frame rate (30 frames

per second) and a low bandwidth (28.8 Kbps), Windows Media Encoder keeps the bandwidth constant, and attempts to achieve the requested frame rate by lowering image quality.

Your CPU can become overburdened, especially in multiple-bit-rate encoding. When the number of tasks per second is too great for the processing power of your CPU, the encoder adjusts to the environment by dropping frames. An occasional dropped frame during a high-action sequence is not that noticeable, but image quality and frame rate can be degraded when your CPU is at 100 percent most of the time. For the best quality, reduce the number of streams, the image size, or the frames per second until usage is no higher than 90 percent.

When encoding from one file to another, Windows Media Encoder adjusts to the processor speed. Because time is not an issue, the encoder takes as long and uses as much CPU bandwidth as necessary to render the digital media without compromising quality. A 30-second file can take five minutes to encode on one machine and 10 seconds on another depending on processor speed. The encoder maximizes use of the CPU to keep encoding time to a minimum. You can see that the CPU is at 100 percent, but unlike live encoding, this does not mean frames are being dropped.

Custom Profiles
This section explains how to set up Windows Media Encoder for live streaming using a profile that you create. On the Windows Media Encoder menu bar, click Session, and then click New Session Wizard. Assuming current encoder settings have been saved, the New Session Wizard appears. Move through the wizard to the Profile Selection screen. For more information about the other screens, see Windows Media Encoder 7 Help documentation. Click Create and Manage Profiles, and then click New. The New Profile wizard opens.

General Information

On the first screen of the wizard, select Compressed, and select the Video check box. Enter a name for your new profile and a description.

Audience Selection

On the next screen, select up to 10 Target Audience check boxes. Select as many as you need to provide your audience with a quality viewing experience, but keep in mind that the more you add, the more CPU processing is required. You can also add a new Target Audience, or edit or remove existing Target Audiences. For each selection you make, a video stream is encoded at the Media Bit Rate indicated.

Figure 2. Audience Selection dialog box

Common Stream Settings

Figure 3. Common Stream Settings dialog box

After you have chosen the streams to encode, select one audio codec (compression type) and one video codec, format, and video size. These settings are used to configure all of the streams. Deciding which settings to enter requires some experimentation and practice. There are no set rules. What you enter depends on the input source, the desired effect, and personal taste-all subjective decisions. But you do have many choices.

Individual Video Stream Settings

On the next screen, enter settings for frame rate, key frame interval, and image quality. Click the tabs to enter settings for each stream. For more information about these settings, see the Help documentation that comes with Windows Media Encoder 7.

Here are some points to consider. If playback appears jerky despite having selected a high frame rate (for example, 15 frames per second), it is most likely due to an imbalance in your Windows Media Encoder settings. For example, if you enter a high frame rate, a large image size, and a bandwidth that is too small, the encoder compensates by not only lowering image quality, but by dynamically adjusting the frame rate. Because there are so many factors involved in creating quality streams, the settings you enter can be modified by the encoder while encoding to limit the streams to the set bandwidths.

The only way to tell if your settings work satisfactorily for a given type of video is by testing them. Before

going live, try different settings. View the results by connecting Windows Media Player directly to the encoder output (http://EncodingMachine:port).

If the frame rate is uneven, lower the rate, or reduce the image size. In Windows Media Player, click View, and then click Statistics. If the Actual rate is much lower than the Frame rate, it can be because the encoder is being forced to drop frames. The playback can appear smoother if you lower the frame rate from 15 to 4. In addition to monitoring playback, check your encoding computer's performance monitor. If CPU usage is at 100 percent constantly, the encoder compensates by dropping frames.

Summary
Intelligent streaming is, for the most part, completely automatic. The interplay between client and server takes place behind the scenes. If you have added multiple-bit-rate streaming, the server can intelligently adjust the bit rate according to the current bandwidth, so the presentation received by the user is smooth.

Intelligent streaming is another step by Windows Media Technologies toward creating the ideal user experience. The goal is to make the experience transparent: the user should only be aware of the content, not the container, and you should only be concerned with creating great content. Intelligent streaming is a major step toward the understanding and management of media presentation over networks. With Windows Media Technologies, you can create one Windows Media file or encode one live stream, and users connected at many different speeds with a multitude of network conditions can enjoy a high-quality presentation. Most importantly, they can enjoy the content without experiencing irritating interruptions and transmission break-up.