Transponder Failure Recovery (Eng)

PBS Business Continuity
&
Disaster Recovery
April 2011
PBS BCP & Disaster Recovery

PBS Distribution Services & Workflow
BCP / DR Initiatives
Recovery Solutions
Satellite / Transponder Recovery
PBS DRS Facility
Do-it-yourself DR plan template
Q&A

Stefan Petrat
VP, Distribution Operations & Engineering
Thomas Crowe
Sr. Director, Technology
James Cutright
Sr. Project Manager Interconnection
Cindy Jackson
Director, Operations Strategic Planning & BCP
PBS Distribution Services
PBS has been providing signals to the member stations from the PBS
Technology Center in Springfield Virginia since 1978.
The PBS Technology Center is comprised of Network Operations (NOC)

and Technical Maintenance (TMC) services.
PBSs Technology Center currently originates and uplinks 11 digital video

services, including two 24-hour a day HD feeds and two HD soft feeds.
These digital services are transmitted to three different satellites over both
Ku and C-Band satellite transponders.
The PBS Tech Center is located 12 miles southwest of downtown

Washington DC near Springfield, Virginia on five acres of land owned by
PBS. Two buildings on campus;
SOC-1 used for NOC & SOC operations
SOC-2 used for TMC, Lab and administration.
4
PBS
Technology Center
SO
C-2
SOC-1 Building
SOC-1
SOC-2 Building
5
Distribution Services & Workflow
Real-time Services
NPS services: HD01, HD02

Soft services: HD03, HD04, SD01
VMe, Create, World: SD02, SD03, SD04
Tape feeds, Ingest & Live Turn-around
Content QC
Satellite Access Coordination
HD04, SD05, SD06, SD07
Disaster Recovery / Business Continuity
Engineering & Lab
Non Real-time
Supporting development & testing
Facilities support
7

PBS Workflow - Overview
Telstar 5
Transponder 25V (KU)
PBS Distribution Ku Band
AMC-21
125W
Ku Band
Vendor: SES World Skies
Satellite EOL: 10/2022
AMC-21 Tx22
AMC-21 Tx23
AMC-21 Tx24
Contract End: 09/2016

Bandwidth: 36Mhz
Modulation: DVB-S2 8PSK
MCPC MPEG2
Fully Protected

Bandwidth: various
Modulation: DVB-S QPSK
SCPC MPEG2
Transponder Protected

Bandwidth: 36Mhz
MCPC MPEG2
Fully Protected
Services
1. HD01
2. HD02
Services
1. HD04 (18Mhz)
2. SD05 (6Mhz)
3. SD06 (6Mhz)
4. SD07 (6Mhz)
Services
1. HD03
2. SD01
3. SD02 (Create)
4. SD03 (VMe)
5. SD04 (World)
AMC-21 Tx20
AMC-21 Tx21
Contract End: 09/2013 & 03/2014

Bandwidth: various
Modulation: various
SCPC

Bandwidth: 36Mhz
SCPC
Services
1. Montana PBS (6Mhz)
2. Oklahoma PBS (10.75Mhz)
Services
1. NRT
Business Continuity
&
Disaster Recovery
Initiatives
10
Evergreen emergency playback unit installed at Nebraska
48 hours of content available on a video server for instant access & availability
Replaced older DVD system and updated program content
Initial Response Team (IRT) Drill
Status: Completed
Completion date: June 2010
Scenario based table top exercise with PBS senior executives

Status: Completed
Completion date: August 2010, another drill targeted for mid 2011
Satellite Phone Test
Annual verification of backup communication system

Status: Completed
Completion date: August 2010
11
NOC / SOC Evacuation Drills
Level 1 Remote control of Tech Center from Crystal City via MOC
Status: Completed & Ongoing
Completion date: June 2010 & September 2010
Level 2 Program origination and satellite distribution from Crystal City via
MOC
Status: Completed & Ongoing
Completion date: November 2010 & February 2011
Satellite Disaster Recovery Plan update
Update, confirm and initiate recovery solution for satellite and/or transponder
failures
Status: Completed
Completion date: December 2010
12
BCP Software / Crisis Communication update
Recovery Planner will text or email alert staff of issues and desired actions
Canned messages have been loaded into the application
Cindy worked with Anne & Jan in Communications to approve messages for various emergencies
Personnel information update (e.g. mobile phone, home phone, contact details
etc.) is in process on a department by department basis
Company-wide testing has taken place to confirm the system functions as expected
T&O and Interactive were done first since they represent a third of the company. Remainder of company completed; All
Staff email from COO with instructions and to reflect importance of task.
Status: Completed
Completion date: February 2011
IT Recovery Solution Project
Review critical business systems & department priorities and needs

Determine next steps and cost for multi-year capx project starting in FY2012
Status: In progress
Completion date: May 2011
13
Recovery Scenarios
14

Normal operations
Telstar 5
Member
Station
15

Evacuation of Tech Center
Telstar 5
Member
Station
16

Loss of Tech Center
Telstar 5
Member
Station
17

Loss of Satellite or Transponder
Telstar 5
Telstar 5
Transponder 25V Transponder
(KU)
25V (KU)
18

Loss of PBS HQ Crystal City
Telstar 5
Member
Station
19

Overview of Future Operations
Telstar 5
Member
Station
Operational
est. CY2012 / 13
(capx FY2012 / 13)
Operational
Q3/Q4-CY2011
(capx FY2011)
20
Satellite / Transponder
Recovery
21
Transponder failure
loss of a single or multiple transponders on a satellite
Satellite failure
loss of at least 13 or more transponders on a satellite
22
AMC-21
125W
Ku Band
AMC-21 Tx22
AMC-21 Tx23
AMC-21 Tx24

Bandwidth: 36Mhz
Modulation: DVB-S2 8PSK
MCPC MPEG2
Fully Protected

Bandwidth: various
SCPC MPEG2

Bandwidth: 36Mhz
MCPC MPEG2
Fully Protected
Services
1. HD01
2. HD02
Services
1. HD04 (18Mhz)
2. SD05 (6Mhz)
3. SD06 (6Mhz)
4. SD07 (6Mhz)
Services
1. HD03
2. SD01
3. SD02 (Create)
4. SD03 (VMe)
5. SD04 (World)
AMC-21 Tx20
AMC-21 Tx21
Contract End: 09/2013 & 03/2014

Bandwidth: various
Modulation: various
SCPC

Bandwidth: 36Mhz
SCPC
Services
1. Montana PBS (6Mhz)
2. Oklahoma PBS (10.75Mhz)
Services
1. NRT
23
AMC-21 - 125o W
Transponder Failure Recovery
Rank
4
AMC-21
AMC-21
(Tx20 V)
Spare Transponder
Montana / Oklahoma
(same polarity)
(Tx21 H)
Spare Transponder
NRT
(same polarity)
(Tx22 V)
Spare Transponder
HD01 & HD02
(same polarity)
(Tx23 H)
Spare Transponder
HD04, SD05, SD06, SD07
(same polarity)
(Tx24 V)
Spare Transponder
HD03, SD01, SD02, SD03, SD04
(same polarity)
For one or more transponder failures, spare components on the space craft are first used.
If spares are not available, PBS has the right to pre-empt another service from their transponder.
Currently there are 8 spare transponders on AMC-21
24
AMC-21 - 125o W
Satellite Failure Recovery
Rank
1
AMC-21
AMC-1
(Tx22 V)
HD01 & HD02
(Tx18V)
HD01 (HD Packaged)
SD01, SD02, SD03, SD04
(Tx24 V)
HD03, SD01, SD02, SD03, SD04
(Tx23 H)
HD04, SD05, SD06, SD07
(Tx16V)
HD03 (HD Soft)
SD05, SD06, SD07,
HD04
Full details of this recovery scenario are available on PBS Connect

Connect > Engineer Compute > Distribution Operations Engineering > Disaster Recover Plan
25
AMC-1 Tx20
103W
C Band
Bandwidth: 18Mhz
MCPC MPEG2
Services
1. PBS C-Band HD & SD
26
AMC-1 - 103o W
Transponder Failure Recovery
Rank
6
AMC-1
AMC-1
(Tx20 H)
Spare Transponder
PBS HD& SD Digital Feed
(same polarity)
27
AMC-1 - 103o W
Satellite Failure Recovery
Rank
6
AMC-1
SES-1
(Tx20 H)
(Tx16 H)
28
Satellite
Band Xpder
Arc
Uplink
Downlink
Freq
Freq
UL
C/Ku (Mhz) L Band (Mhz) BW (Mhz) Pol
Mod
Type
FEC
Sym Rate
MS/s
Services
AMC-21
Ku
20
125 W
14404.00
1354.00
DVB-S
QPSK
3/4
4.4444
AMC-21
Ku
20
125 W
14412.50
1362.50
10.75
DVB-S2
8PSK
3/4
8.916
Oklahoma (OETA) U/L
AMC-21
Ku
21
125 W
14420.00
1370.00
36
DVB-S2
UNK
8PSK
UNK
NRT Test
AMC-21
Ku
22
125 W
14440.00
1390.00
36
V
H
DVB-S
QPSK
3/4
30.000
3/4
14.028732
3/4
4.444
SD05 SCPC
3/4
4.444
SD06 SCPC
3/4
4.444
SD07 SCPC
AMC-21
Ku
23
125 W
14450.00
1400.00
18
18
AMC-21
Ku
23
125 W
14463.00
1413.00
AMC-21
Ku
23
125 W
14469.00
1419.00
AMC-21
Ku
23
125 W
14475.00
1425.00
AMC-21
Ku
24
125 W
14480.00
1430.00
36
36
DVB-S
QPSK
3/4
30.000
SES-1
16
101 W
6245.00
1130.00
36.00
Analog
N/A
N/A
AMC-1
20
103 W
6316.00
1059.00
18
DVB-S
QPSK
3/4
14.028732
DVB-S
QPSK
DVB-S
QPSK
DVB-S
QPSK
DVB-S
QPSK
Montana PBS
MCPC - HD01 & HD02

HD04
MCPC - HD03, SD01, SD02, SD03 & SD04

Not used
PBS C-Band HD & SD (Consumer use)
29
Network Operations Center
Disaster Recovery Site (DRS)
30

Purpose:
The purpose of disaster recovery/business resumption project is to assure
continuity of Network and Non-Realtime distribution operations needed to
support critical system functions. The business resumption plan should aim at
achieving a systematic and orderly resumption of all operational services. The
plan should provide for a hot standby functionality between sites as well as
restoring full service due to catastrophic failure as soon as possible.
31

Business Decision:
Given the PBS Satellite & Broadcast Operations Disaster Recovery
Planning Guide, and the Integrated Public Alert and Warning System
(IPAWS) that mandates a geographically separated facility, the decision to
build and operate a DRS site seems to have been embraced. Also, as can be
gleaned from the two papers, PBS has a business responsibility to its member
stations and also the Federal Government in its IPAWS commitment.
32

Project Scope:
One of the major challenges facing the Public Broadcasting Service (PBS)
today is preparing for the continuity of the system business following a possible
catastrophic event that disables the operation of the Network Operations
Center (NOC) facility.
Whether referred to as disaster recovery or business continuity, such a
contingency plan must be flexible enough to accommodate both short and longterm outages of the primary transmission site. At the same time, plans to
survive an unlikely catastrophic event must also make sense within the
economics of an ongoing business. .
33

Project Scope:
Business considerations
- What Do We Stand to Lose
- How closely must the backup site replicate our primary site
- What types of recovery scenarios are to be covered
Logistical
- Staffing (UNL-TV, PBS)
- Remote Operation (MOC, NOC)
Operational requirements
- File Content Delivery & Sizing of Content Requirements
- Control Data & Transfer Solution Component
Technologies
34

Technology Solutions:
1. Conventional apparatus
2. IT/Software solution
Objectives for the DRS focused on:
Lower capital cost as compared to NOC

Simplicity of operation, including system compatibility of content files with NRT
Lower operational cost
Lower annual support costs
Side benefit:
Confirmation of new technology for future upgrade of NOC site
35
36

DRS Equipment Center (DEC)
is a full function technical equipment center
contained within the UNL-TV Nebraska facility. All
equipment needed to support the DRS operation is
contained within the DEC. The DEC operates on its
own power (commercial & emergency generator)
with full Uninterruptable Power Supply (UPS).
Redundant HVAC units provide the necessary
cooling and humidity control for the equipment and
systems housed within the DEC. The equipment
center is protected by a staged fire alarm system
insuring the safety of systems and personnel.
Within the DEC are equipment racks, work stations,
tape storage shelves, cable trays, electrical
distribution, grounding connections and access to
UNL-TVs cable trays which connect to their
technical equipment center and satellite antenna
farm.
37

PBS DRS Remote Operations position 1 & 2
(PDRO1 & PDRO2)
Is a pair of remote operations positions located at
the PBS NOC (PDRO1) in Alexandria, Va. and the
PBS MOC (PDRO2) in Arlington, Va. Both
positions will have full remote control of the DRS
facility. PDRO1 will operate the DRS facility on a
daily basis with PDRO2 being available in the event
the NOC is disabled. In the event of an actual
disaster involving the NOC, PDRO2 at the MOC will
assume daily operations of the DRS Facility
38

DRS Network Operations Center (DNOC)
Is a full function operations facility located within the
UNL-TV Transmission Center. It is the local
operations, status, monitoring, alarm & control
position in Lincoln, Ne. It will be the operations
position during scheduled training operations,
monthly satellite tests, and PBS Network
Operations Center (NOC) maintenance or in the
event of an actual disaster and the PBS NOC is
disabled (either long term or short term).
Laptop w/ Broadband & VPN Access

Is a full function operations workstation with full
access to all applications and monitoring within the
DRS. In a worse case scenario the DRS may be
operated in this manner.
39

Terrestrial Interconnection Sub-system (TIS)
Means dedicated and internet based connectivity
between the Media Operations Center (MOC) in
Arlington, Va., the Network Operations Center
(NOC) in Alexandria, Va. and University of
Nebraska Lincoln Television Department (UNLTV) in Lincoln, Ne
40

Ku Band Satellite Uplink Transmission Subsystem (KSUTS)
Means an RF transmission system, housed within
the UNL-TV satellite equipment shelter located in
the backyard dish farm, consisting of DVB-S/S2
QPSK/8PSK modulation sub-systems, IF-RF upconversion sub-systems, high power amplifier subsystems, automatic fault protection sub-systems,
signal switching sub-systems, remote
monitoring/control/alarm sub-systems,
interconnected to satellite uplink antennas
(including redundant antenna). We have exclusive
use of this uplink chain and antenna, there will
never be a conflict with UNL-TV?
41

C Band Satellite Uplink Transmission
Sub-system (CSUTS)
Means an existing UNL-TV RF
transmission system (fiber STL from UNLTV Studio Building), contained within the
UNL-TV satellite equipment shelter located
near the Lincoln Municipal Airport (LNK)
(including redundant antenna).
42

Real Time (RT) Program Origination Subsystem (POS)
Means a platform of equipment and software
including but not limited to a traffic system,
integrated ingest & play to air systems, signal
processing sub-systems, SDI & IT switches &
routers and IT management tools enabling the
delivery of RT program streams to Public
Television from UNL-TV Ne.
Cache-Archive Sub-system (CAS)

Means a disk based cache & LTO tape
based archive sub-system.
43

Non-Real Time (NRT)
File Transfer System (FTS)
enabling the delivery of NRT program files to Public
Television Stations from UNL-TV Ne
44

Satellite Encoding/Mux Sub-system (SEMS)
Means a master control system that is designed
to encode and multiplex multiple channels of SD
and HD video into transport streams for
distribution to Public Television Stations from
UNL-TV Ne
Local & remote Status, Monitoring, Alarm &

Control sub-system (SMAC)
enabling the control and monitoring, both locally
and remotely, of the systems and services
originating from or being distributed by the DRS
facility
45

The following conditions are examples of DRS activation scenarios:
NOC Diversity: In order to maintain PBS programs and services to PTV
Stations the DRS will provide satellite uplink services whenever the NOC
uplink is disabled by 1) sun transients, 2) rain-fades, 3) blizzard conditions, or
any other naturally occurring phenomena.
NOC Maintenance: In order to maintain PBS programs and services to PTV
Stations the DRS will provide program origination and uplink services
whenever the NOC facility is out of service for maintenance functions
(scheduled event).
NOC Emergency: In order to maintain PBS programs and services to PTV
Stations the DRS will provide program origination and uplink services
whenever the NOC facility is temporarily (short term) out of service due to an
emergency condition.
46

The following conditions are examples of DRS activation scenarios:
NOC Disaster: In order to maintain PBS programs and services to PTV
Stations the DRS will provide program origination and uplink services if the
NOC facility is rendered inoperable due to a catastrophic event.
MOC/NOC Disaster: In order to maintain PBS programs and services to PTV
Stations the DRS will provide content acquisition and management, program
origination and uplink services if both the MOC and NOC facilities are
rendered inoperable due to a catastrophic event.
47

Past & Present: A Project History
Original Concept:
Limited services (2 to 3 channels)

Legacy technology (automation, video servers, branding, )
Parallel operation (in sync with NOC)
Restoration of supported services in < 15 mins
Remote control from PBS (NOC only)
Media based file delivery (LTO tape)
AS03 file format for distribution
Limited connectivity for:
Urgent file delivery only (limited by bandwidth)
Remote control
Status and monitoring
System integrated into existing PTV facilities
48

Site Selection:
RFP written and issued to PTV

PTV Stations responded
Responses review/evaluated
Site selected (UNL-TV, Lincoln, Ne)
Technology Changes:
IT based Ingest & Play to Air systems emerged

Integrated systems on the market (multi-vendor)
Asset management included in integrated solutions
SD & HD capability
Client/Server architecture (Directly Supports Workstation Model)
Web based services
49

Technology Changes (Cont.):
Transcoding/Re-wrapping on IT server platforms comes of age

Lower initial investment as well as reduced operating costs
New Concept for DRS:
Support the full bouquet of NOC services

HD01, HD02, HD03, HD04, SD01, SD02, SD03, SD04, SD05, SD06
Use DRS Uplink as Diversity Site

Rainfade, Sun Transient,
Use IT technologies where applicable

S/W versus H/W solutions
Play to Air, Ingest, Cache, Convert
Use commodity versus proprietary H/W (PBS purchases H/W)

IT server platforms (multi CPU, multi-core, local cache)
Commodity video cards (multi-vendor)
IT network switches, routers, VPN,
50

New Concept for DRS:
Client/Server architecture
Stream Monitoring Solution back to PBS (Multiviews, Desktops, )
Workstation based operations (Desktop/Laptop)
DRS Lab at PBS (support S/W & H/W testing prior to implementation)
Program Origination Subsystem (POS) Vendor Shoot-out:
DRS Design Document created

Proof of Concept Lab @ PBS (concepts tested and demonstrated)
RFB issued to vendors
Multi-vendors responded
Three vendors participated in the Shoot-out
Two weeks allocated per vendor (1 wk setup, 1 wk test/eval)
51

Program Origination Subsystem (POS) Vendor Shoot-out:
Demo/Evaluation conducted in Lab @ PBS, Alexandria

Results evaluated and scored
Vendors evaluated based on:
Scored performance,
Compliance to DRS model (design specification),
and business parameters (D&B, assets, )
Vendor selected for DRS
GmbH, Munich, Germany
Cache/Archive Subsystem (CAS):
Centralized Disk Cache & Tape Archive Solution

Common to RT & NRT Systems
Commodity H/W based solutions
52

Cache/Archive Subsystem (CAS):
High Availability and Reliability Required

No standard off the shelf solution
Develop Solution with Vendor to support DRS
Test/Eval Proposed Solution in PBS Lab
PBS DRS Connectivity (TIS):
Determine Bandwidth Requirements

Connectivity between MOC and DRS (File Delivery, Status/Mon/Alarm/Ctl)
Connectivity between NOC and DRS (Transport Streams, Status/Mon/Alarm/Ctl)
Evaluate, Test, and Demonstrate Available Solutions
Determine Best Quality Solution for DRS (BW vs $$$)
53

More new Technology for DRS:
IP based Status, Monitoring, Alarming (TS over IP)

Research Available Applications
Implement S/W Solutions where possible
Commodity H/W where applicable
Exception based monitoring/alarming
Decode SPTS and MPTS within Mon/Alarm Engines
Full support for Closed Captions & Annc Data
DVB Encode/Multiplex
Use TSoIP for Signal Transport
Use Ethernet (cabling, switches, vlans, ) to:
Interconnection (Encoder/Mux/Modulator)
Provide Full Redundancy
Support TSoIP Status/Monitoring/Alarming
Support Full Control & Configuration
Encode using S/W where viable
Multiplex using S/W where viable
54

Status, Monitoring, Alarming and Control:
Local Monitoring, Alarming & Control

Remote Monitoring, Alarming & Control
Duplicate User Experience at PBS (MOC, NOC)
Support Baseband, ASI & TSoIP formats
Exception Based
Display HD/SD, Alarm on Faults
Display Audio bouquet (16 chs), Alarm on Faults
Display Closed Captions, Alarm on Faults (608/708)
Display Annc. Data (AFD, V-Chip, ), Alarm on Faults
Fault Tolerant Solution
55

MPEG4 Requirements:
System must be MPEG4 Ready

Work with PBS MPEG4 Team to Eval/Demo/Test Systems
RF System for DRS:
Design Signal System in Cooperation with UNL-TV

Use Existing Regional Uplink Infrastructure in Design
Integrate Additional Equipment & Systems to Support DRS
Design Control System Using Existing UNL-TV Technologies
Design for Joint Use of Uplink for DRS & Regional Services
Activate Diversity Site functionality as soon as possible
56

Where we are at this time:
The Agreement with UNL-TV is in place

The General Contractor has been hired
DRS Equipment Center (DEC) construction underway
The Program Origination (POS) Vendor is under Contract with PBS
Pre-build of the POS Subsystem Racks at PBS is underway
Connectivity Solutions are being Tested/Evald at this time
RF Design in Support of Diversity Site Functionality is underway
We are in development for the Redundant Cache/Archive Subsystem (working w/
Vendors)
Working with the MPEG4 Team, Test/Eval/Demo Vendor Solutions
57

Project Goals:
Diversity Site Capability no later than end of Q2, beginning of Q3

Power Up Full DRS no later than Q3 of 2011
Commence Staff Training (PBS & UNL-TV) in Q3, 2011
Parallel Operations no later than end of Q3, 2011
First Satellite Testing and Demonstration, during Q4, 2011
DRS Facility transferred to PBS/UNL-TV Operations, end of Q4
Project wrap-up and documentation completed by end of Q4, 2011
58

Thanks to All of You !!!

Project Partners:
NGIS Project, The Grand Pobah!, Jerry Butler, Alexandria, Va
PBS Engineering & Operations, Stefan Petrat, Alexandria, Va

UNL-TV, Stacey Decker, Lincoln, Nebraska
HA Design Group, Consulting, Willy Halla, Springfield, Va
Caleca Consulting, MPEG4, Ed Caleca, Washington, DC
Burt Hill Architects, Scott Jones, Washington, DC
Cinegy GmbH, POS, Jan & Daniella Weigner, Munich, Germany
ABC Electric, Inc., General Contractor, Brad Wittstruck, Lincoln, Ne

JT Packard, UPS/GenSet, Dani Maher & Sam Shannon, Verona, WI
More to Come!!!
59
DIY
Disaster Recovery Template
60

Do-it-Yourself DR Template
Organization of Document
Emergency Management Plan
Disaster Recovery Plan
How-To Document
Foundational Document
Building block for a complete business continuity plan
Published in PBS Connect (May 2010)

Engineer / Compute Technical Reference Library
61

EMERGENCY MANAGEMENT PLAN
Emergency Management is pre-determined actions that are

performed immediately following emergency responders to an
incident and before recovery. It is designed to create a state of
readiness for an organization to function and begin a strategic,
orderly recovery, avoiding confusion and reducing exposure to error.
It is also known as Crisis Management.
Emergency Management Plan
Crisis Manager role and responsibilities
Emergency Teams roles and actions to perform
Initial Response Team

Emergency Management Team
Logistics Support Team
Emergency Response Team
62
Team Activation During a Crisis

INCIDENT
Activate Emergency Procedures
Activate Initial Response Team
Building Evacuation (ERT). Security

Notified. Civil Authorities Notified.
Damage Assessment. Disaster Declaration. Alert
Critical Vendors. Establish Emergency Mgmt Ctr
Activate Emergency Mgmt Team
Recovery
Recommendations.
Required Resources.
Manage Business Unit
Activate Business Unit Recovery Teams
Activate Logistics Support Team
Recovery of respective
business areas
Support IRT and EMT for recovery of the organizational

infrastructure.
63

Do-it yourself DR Template
64

Disaster Scenarios
Critical Functions
Checklists
Appendices
65

DISASTER SCENARIOS
Fire/Explosion
Loss of Tower or Transmitter
Loss of Satellite
Loss of STL
Satellite Uplink Failure
Loss of PBS downlink or NGIS
server
Loss of electrical power
Loss of HVAC
Internal Hazardous Material
External Hazardous Material
Weather Advance Notice

Weather No Notice
Loss of Transportation
Loss of Telecom
IT Disaster
Workplace Violence
Flooding
Civil Unrest
Loss of Multiple Key Personnel
66

CRITICAL FUNCTIONS
Employee Safety
Executive Decision Making
Program Production
News Production
Program Origination
Program Distribution
Program Distribution
Over-the-Air Transmission
PBS Downlinking / Storage
Internal and External Communications
Feeds to cable, DirecTV, Dish, etc.
67

CHECKLISTS
One for each Scenario

Indicates how to recover all critical functions
Less than 2 pages in length
Gives all pertinent information but not all details
Refers to Appendices with more details if necessary
Split into two timeframes: 1st 72 hours and afterwards

Some scenarios use the same checklist
68

APPENDICES
Evacuation Plan
Emergency Communications
Emergency Operations Centers
Shelter-in-Place
Television Ops Restoration
News Production Restoration
Studio Production Restoration

Long-term Evacuation
Electrical load reduction guidelines
Vendor List
IT Restoration
69

How-To Document
PLAN DEVELOPMENT PROCESS
Process is described in the

How-To Document
How-To Document also
includes:
Risk Assessment Guidelines
Best Practices information
from other station plans
Typical Recovery Solutions
70
Questions?
71

Transponder Failure Recovery (Eng)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Transponder Failure Recovery (Eng)

Uploaded by

Copyright:

Available Formats

PBS Business Continuity

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS Distribution Services

The PBS Technology Center is comprised of Network Operations (NOC)

PBSs Technology Center currently originates and uplinks 11 digital video

The PBS Tech Center is located 12 miles southwest of downtown

PBS BCP & Disaster Recovery

Distribution Services & Workflow

PBS Distribution Services

NPS services: HD01, HD02

Tape feeds, Ingest & Live Turn-around

Satellite Access Coordination

HD04, SD05, SD06, SD07

Disaster Recovery / Business Continuity

Engineering & Lab

Supporting development & testing

PBS Distribution Services

PBS Distribution Ku Band

Contract End: 09/2016

Contract End: 09/2016

Contract End: 09/2016

Contract End: 09/2013 & 03/2014

Contract End: 09/2016

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

Evergreen emergency playback unit installed at Nebraska

Initial Response Team (IRT) Drill

Scenario based table top exercise with PBS senior executives

Satellite Phone Test

Annual verification of backup communication system

PBS BCP & Disaster Recovery

NOC / SOC Evacuation Drills

Satellite Disaster Recovery Plan update

PBS BCP & Disaster Recovery

BCP Software / Crisis Communication update

Canned messages have been loaded into the application

IT Recovery Solution Project

Review critical business systems & department priorities and needs

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

PBS BCP & Disaster Recovery

Contract End: 09/2016

Contract End: 09/2016

Contract End: 09/2016

Contract End: 09/2013 & 03/2014

Contract End: 09/2016

HD01 & HD02

HD04, SD05, SD06, SD07

HD03, SD01, SD02, SD03, SD04

Full details of this recovery scenario are available on PBS Connect

PBS BCP & Disaster Recovery

PBS HD& SD Digital Feed

PBS HD& SD Digital Feed

PBS HD& SD Digital Feed

PBS BCP & Disaster Recovery

Oklahoma (OETA) U/L