Networking for Games

Mitchell Keith Bloch

Material taken from Sugih Jamin and John Laird University of Michigan 2260 Hayward Street Ann Arbor, MI. 48109-2121 bazald@umich.edu

October 26, 2011

Mitchell Keith Bloch (University of Michigan)

Networking for Games

Introduction

Why multiplayer games? People are better at most strategy than current AIs People are less predictable Can play with people, communicate in natural language Add social aspect to computer games Provides larger environments to play in, with more characters Make money as a professional game player

Mitchell Keith Bloch (University of Michigan)

Networking for Games

1 / 38

Representative Game Types

Head-to-head death-match:
Fast-pace, intense interaction/combat No persistent state Players form ad-hoc, short-lived sessions Any client can be a server Requires matchmaking service: built-in lobby or use GameSpy Examples: X/NetTrek (1980s, simulation), Doom (1990s, FPS), Counter-Strike, StarCraft, AoE, etc. (RTS-combat)

Persistent-world, massively multiplayer online game (MMOG)

Mitchell Keith Bloch (University of Michigan)

Networking for Games

2 / 38

Basics

Outline

Basics Latency TCP/IP and UDP/IP Topology Distributed Computing and Latency

Mitchell Keith Bloch (University of Michigan)

Networking for Games

Basics

OSI Model

1 2 3 4 5 6 7

Physical Layer Data Link Layer Network Layer How we address hosts (IPv4/IPv6) Transport Layer What we build on (TCP/UDP) Session Layer Presentation Layer Where we get security (SSL/TLS) Application Layer What we develop

We care most about TCP/UDP and the application layer.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

3 / 38

Basics

Application Layer

Limitless sublayers exist within the application layer:

1 2 3 4 5 6 7

Direct IP connections between players Game server for connecting players together Tracker / match making service to connect players to servers Tracker aggregator (aggregator (...)) Download / patch server Store Store aggregator (aggregator (...))

We care most about 1, somewhat about 2, and just a little bit about 3.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

4 / 38

Basics

In-Game vs Backend
Differentiate between in-game networking and backend infrastructure Backend infrastructure (see Yahn Berniers talk at GDC2000 rtsp://media.cmpnet.com/gamasutra/bernier.rm): Lobby where gamers meet Authentication and CD key checking Accounting and billing Ranking and ladder Reputation and black list Buddy lists, clans, and tournaments Mods and patches management Virtual economy Beware of DDoS Issues: scalability, adapting to failure, security
Mitchell Keith Bloch (University of Michigan) Networking for Games 5 / 38

Basics

Concerns

Bandwidth
Too little bandwidth means we cant get enough data per unit time

Latency
Too high latency means we cant get data in time for it to be useful

Computational cost
Too high computational cost means we cant manage enough connections Or we cant initialize connections fast enough

Which do you think were most concerned about? Hint: Everyone advertises bandwidth, and youre not creating an MMO.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

6 / 38

Basics

Bandwidth Limitation
What is bandwidth? What information is sent? Depends on your computing model, distributed object or message passing Game state: coordinates, status, action, facing, damage User keystrokes Commands/moves For AoE: 1 every 1.5-2 sec, up to 3-4 commands/sec during battles (but some of these are redundant and can be ltered out) Current lower limit assumed: 56 Kbps Broadband

Mitchell Keith Bloch (University of Michigan)

Networking for Games

7 / 38

Basics

Bandwidth Limitation Continued

Bandwidth requirement has been HIGHLY optimized Even with audio chat, takes up at most 8 Kbps So, bandwidth is not a big issue (but note the asymmetric nature: at 8 Kbps, you can only support a total of 4 players on a 28.8 Kbps modem) Must be continually vigilant against bloat HOWEVER, with player-created objects and worlds, bandwidth becomes an issue again: use streaming, levels of details, and pre-fetching

Mitchell Keith Bloch (University of Michigan)

Networking for Games

8 / 38

Latency

Outline

Basics Latency TCP/IP and UDP/IP Topology Distributed Computing and Latency

Mitchell Keith Bloch (University of Michigan)

Networking for Games

Latency

Latency

Recall that latency is the time between when the user acts and when they see the result Latency is arguably the most important aspect of a game network
Too much latency makes the game-play harder to understand because the player cannot associate cause and effect It makes it harder to target objects (the lead becomes to large) There is signicant psychological research on this topic

Latency is not the same as bandwidth

A freeway has higher bandwidth than a country road, but the speed limit, and hence the latency, can be the same Excess bandwidth can reduce the variance in latency, but cannot reduce the minimum latency (queuing theory)

Mitchell Keith Bloch (University of Michigan)

Networking for Games

9 / 38

Latency

Latency Limitation
How is latency different from bandwidth? Latency: RTS: 250 ms not noticable, 250-500 ms playable, > 500 ms noticable FPS: 150 ms preferred Car racing: < 100 ms preferred, 100-200 ms sluggish, 500 ms, car out of control Players expectation can adapt to latency It is better to be slow but smooth than to be jittery

Mitchell Keith Bloch (University of Michigan)

Networking for Games

10 / 38

Latency

Sources of Latency
Consider a client sending and receiving data from a server There are four sources of latency in a game network
Frame rate latency: Data only goes out on or comes in from the network layer once per frame, and user interaction is only sampled once per frame Network protocol latency: It takes time for the operating system to put data onto the physical network, and time to get it off a physical network and to an application Transmission latency: It takes time for data to be transmitted to the receiver Processing latency: The time taken for the server (or client) to compute a response to the input

You cannot make any of these sources go away

You dont even have control over some of them Remember Amdahls law when trying to improve latency
Mitchell Keith Bloch (University of Michigan) Networking for Games 11 / 38

Latency

Reducing Latency Part I

Frame rate latency:

Increase the frame rate (faster graphics, faster AI, faster physics)

Network protocol latency:

Send less stuff (less stuff to copy and shift around) Switch to a protocol with lower latency But may have impact on reliability and security

Transmission latency:
Send less stuff less time between when the rst bit and the last bit arrive Upgrade your physical network (from dial-up to DSL, for instance)

Mitchell Keith Bloch (University of Michigan)

Networking for Games

12 / 38

Latency

Reducing Latency Part II

Processing latency:
Make your server faster Have more servers

The sad fact is, networking researchers and practitioners are almost never concerned with latency
Most applications can handle higher latency (who else cares about latency?) When did you last hear a DSL/Cable add that promised lower latency? Possibly changing with concern over buffer bloat!

Mitchell Keith Bloch (University of Michigan)

Networking for Games

13 / 38

Latency

Working with Latency

If you cant get rid of latency, you can try to hide it Any technique will introduce errors in some form - you cannot provide immediate, accurate information Option 1: Sacrice accurate information, and show approximate positions
Ignore the lag and show a given player old information about the other players Try to improve upon this by guessing where the other players are. But if your guess is wrong, incorrect information is shown

Option 2: Sacrice game-play:

Deliberately introduce lag into the local players experience, so that you have enough time to deal with the network

Mitchell Keith Bloch (University of Michigan)

Networking for Games

14 / 38

TCP/IP and UDP/IP

Outline

Basics Latency TCP/IP and UDP/IP Topology Distributed Computing and Latency

Mitchell Keith Bloch (University of Michigan)

Networking for Games

TCP/IP and UDP/IP

Why TCP or UDP?

TCP
TCP is reliable, guaranteeing in-order delivery (or disconnection) TCP has no size limitations for how much data can be sent in one transmission TCP plays nice with the network Generally considered easier to get right

UDP
UDP is fast UDP has no built-in cost for connecting to multiple hosts UDP doesnt play nice with the network Generally considered faster

Mitchell Keith Bloch (University of Michigan)

Networking for Games

15 / 38

TCP/IP and UDP/IP

TCP vs UDP: More Technical

IP routes packet from source to destination, max IP packet size is 64 KB, may be fragmented What TCP (Transmission Control Protocol) gives you:
Reliable delivery Retransmission and reordering Congestion control

What UDP (User Datagram Protocol) gives you:

Unreliable delivery No retransmission, packets not ACKnowleged, no reordering No congestion control So, more or less, plain IP service

Mitchell Keith Bloch (University of Michigan)

Networking for Games

16 / 38

TCP/IP and UDP/IP

Which to use?
Game requirements: Late packets may not be useful anymore Lost information can sometimes be interpolated But loss statistics may be useful Use UDP in game: Can prioritize data Can perform reliability if needed Can lter out redundant data Use soft-state Send absolute values, not deltas Or if deltas are used, send baseline data periodically Must do congestion control if sending large amount of data
Mitchell Keith Bloch (University of Michigan) Networking for Games 17 / 38

TCP/IP and UDP/IP

TCP
Server:
1 2 3

Open a TCP port to listen on For each incoming connection, open a new TCP socket Send and receive bytes from TCP sockets, maintaining sufcient state to handle incompletely received data Possibly do a DNS lookup to learn a host IP address Open a new TCP socket to connect to a known host IP & port Send and receive bytes from TCP sockets, maintaining sufcient state to handle incompletely received data

Client:
1 2 3

Note that you may not receive all the data you need at once. If attempting to stay interactive, you have to be able to keep your game running and try to get the rest of the data later. This bookkeeping can be tricky.
Mitchell Keith Bloch (University of Michigan) Networking for Games 18 / 38

TCP/IP and UDP/IP

UDP
Server:
1 2

Open a UDP port to listen on For each incoming packet, decide what to do based on the IP address, port, and data Send packets to selected address/port combinations Open a UDP port to listen on For each incoming packet, decide what to do based on the IP address, port, and data Send packets to selected address/port combinations

Client:
1 2

Note that there is no longer a concept of a listener port. There is also only one socket shared between all connections. However, reliability is lost. Ensuring that data from older packets not interfere with data from newer packets requires work, and it is impossible to detectonly disconnection with certainty.
Mitchell Keith Bloch (University of Michigan) Networking for Games 19 / 38

TCP/IP and UDP/IP

Reliable UDP
UDP doesnt provide reliability, write your own reliable udp for moves that must be reliable, e.g., sniper shots Desirable features: Error control: do checksum Ordering: use sequence # Reliability: acknowledge packet (use cumulative ACK), retransmit if not ACKed, timeout value a function of average rtt (round-trip time) Flow control: dont send more than the target can handle; use stop-and-wait or sliding-window If you need all of these, all of the time, just use TCP. It works well.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

20 / 38

TCP/IP and UDP/IP

Serialization

Required for both TCP and UDP

Must create a protocol for putting data into the pipe and getting it back out (TCP) Must create a protocol for putting data into packets and getting it back out (UDP)

Must decide carefully what data must be sent

Send all data, or some subset Send precise data, or truncated data Track receipt, or assume adequate receipt (UDP)

Mitchell Keith Bloch (University of Michigan)

Networking for Games

21 / 38

TCP/IP and UDP/IP

In-Game Networking

Networking topology: client-server vs. peer-to-peer Computing model: distributed object vs. message passing Which protocol to use? tcp, udp, reliable udp Bandwidth limitation Latency limitation Consistency Cheat proong Socket programming

Mitchell Keith Bloch (University of Michigan)

Networking for Games

22 / 38

Topology

Outline

Basics Latency TCP/IP and UDP/IP Topology Distributed Computing and Latency

Mitchell Keith Bloch (University of Michigan)

Networking for Games

Topology

Peer-to-Peer

Peer-to-peer with O(N 2 ) unicast connections: Each player is connected directly to all other players Each player simulates the whole world Advantages: reduced latency, no single point of failure Disadvantages: easier to cheat, not scalable: each client must send and receive N 1 messages Used in Age of Empire

Mitchell Keith Bloch (University of Michigan)

Networking for Games

23 / 38

Topology

Client-Server

Two avors:
Ad-hoc servers: death match Dedicated servers: MMORPG

Two types of clients:

Clients simulate world, server has authoritative state: allows for client-side dead reckoning (QuakeIII/Half-Life). Clients for I/O, all simulations at server: useful for thin clients, e.g. cell phones, and persistent-world MMOG.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

24 / 38

Topology

Client-Server rather than P2P

Advantages: each client sends only to server, server can aggregate moves Advantages (dedicated servers): cheat-proong, server can be better provisioned, persistent states (for MMOG) Disadvantages: longer delay, server bottleneck, single point of failure, needs server management

Mitchell Keith Bloch (University of Michigan)

Networking for Games

25 / 38

Topology

MMOG Architectures

The world replicated at each server (shard); each shard contains an independent world; players go to specic shard
Most MMORPG

The world replicated at each server (mirror); all the worlds are synchronized; players see everyone across all mirrors
Mirrors must be kept consistent

The world is split up into regions, each region is hosted by a different server
Second Life is a great example Mirrors must be kept consistent

Mitchell Keith Bloch (University of Michigan)

Networking for Games

26 / 38

Topology

Distributed Computing Model

Usually your game company will have its preferred computing model and would provide high-level libraries to implement the model Distributed objects: Characters and environment maintained as objects Player inputs are applied to objects (at server) Changes to objects propagated to all players at end of game loop Object update usually implemented as one or more library calls

Mitchell Keith Bloch (University of Michigan)

Networking for Games

27 / 38

Topology

Distributed Computing Model

Message passing: Player inputs (either button pushes or higher-level movements) are sent to other players (or server) All players update their own game state Or server updates the global game state and send it out to all players

Mitchell Keith Bloch (University of Michigan)

Networking for Games

28 / 38

Distributed Computing and Latency

Outline

Basics Latency TCP/IP and UDP/IP Topology Distributed Computing and Latency

Mitchell Keith Bloch (University of Michigan)

Networking for Games

Distributed Computing and Latency

Lock-Step

Each player receives all other players moves before rendering next frame Problems: Long Internet latency Variable latencies Speed determined by the slowest player

Mitchell Keith Bloch (University of Michigan)

Networking for Games

29 / 38

Distributed Computing and Latency

Bucket Synchronization

Buffer both local and remote moves Play them in the future Each bucket is a turn, say for about 200 ms Bucket size can be adapted to measured rtt Problems: What if a move is lost or late? Game speed (bucket size) determined by slowest player

Mitchell Keith Bloch (University of Michigan)

Networking for Games

30 / 38

Distributed Computing and Latency

Consistency

For consistency ALL user input MUST pass through the synchronization module Be careful with random number generators. Isolate the one used for game-state updating from other uses (ambient noise etc.) Design for multiplayer from the start. Single-player becomes a special case of single-client multiplayer game.

Mitchell Keith Bloch (University of Michigan)

Networking for Games

31 / 38

Distributed Computing and Latency

Pessimistic Consistency
Every player must see the EXACT same world AoE/AoK/AoM: Uses bucket synchronization Each player sends moves to all other players Each player simulates its own copy of the world All the worlds must be in sync. A designated host collect measured rtt from all players and set future bucket size Problems: Variable latencies Speed determined by the slowest player

Mitchell Keith Bloch (University of Michigan)

Networking for Games

32 / 38

Distributed Computing and Latency

Dead Reckoning

Dead reckoning uses prediction to move objects about even when their positions are not precisely known, reducing the appearance of lag
Each client maintains precise state for some objects (e.g. local player) Each client receives periodic updates of the position of everyone else, along with velocity information, and maybe acceleration On each frame, the non-local objects are updated by extrapolating their most recent position using the available information

With a client-server model, each player runs their own version of the game, while the server maintains absolute authority

Mitchell Keith Bloch (University of Michigan)

Networking for Games

33 / 38

Distributed Computing and Latency

Fixing Extrapolation Errors

What do you do when using dead reckoning, and a new position arrives for another player?
The position that just came in will not agree with the place you have the object, due to extrapolation errors

Two options:
Jump to the correct position Interpolate the two positions over some period
Path followed will never be exact, but will match reasonably well

Mitchell Keith Bloch (University of Michigan)

Networking for Games

34 / 38

Distributed Computing and Latency

Optimistic Consistency with Roll-Back

Observation: dead reckoning doesnt have to be limited to lost packets! Half-Life: Each client plays back its own moves immediately and send the moves to server Each client also dead reckons the other players moves Server computes world and sends its authoritative version to all clients Clients reconcile dead reckoned world with servers version Can result in some jerkiness and perception of shooting around corner Only need to synchronize important events, but must be careful that dead reckoning error doesnt get compounded over time
Mitchell Keith Bloch (University of Michigan) Networking for Games 35 / 38

Distributed Computing and Latency

Best Practices
For smoother playback, decouple bucket size from frame rate (even AoE does this) Immediately render local moves Modify game design to allow for latency and loss, e.g.: Make players wait for elevator Teleportation takes time Require multiple hits per kill Let bullet/missile have ying time Build in inertia, dont allow sudden change in facing

Mitchell Keith Bloch (University of Michigan)

Networking for Games

36 / 38

Distributed Computing and Latency

Reducing Consistency Check

Do area-of-interest management (a.k.a. relevance ltering): Aura: how far you can be sensed (cloaked ships have 0 aura) Nimbus: how far you can sense (use quantum-sensor to detect cloaked ships) Only perform consistency check between objects whose aura and nimbus overlap

Mitchell Keith Bloch (University of Michigan)

Networking for Games

37 / 38

Distributed Computing and Latency

Cheating
AoE doesnt need cheat-proong because each player simulates each move in lock step. All moves are simulated, not just collisions. Half-Life synchronizes only collisions, higher probability for cheating. Cheats (more at megagames.com): Superhuman cheat: auto-aim, auto-position Game-state editing: boost players prole Rule bending: see/walk through walls Sixth-sense cheat Lookahead cheat: claim to be behind slow link Suppress-correct cheat: exploit dead-reckoning claim that own moves were lost, then reconstruct advantageous moves based on others moves
Mitchell Keith Bloch (University of Michigan) Networking for Games 38 / 38