Professional Documents
Culture Documents
Universitat Stuttgart
Institut f i r Parallele und Verteilte Hochstleistungrechner
{ rothermel, wahl}@informatik.uni-Stuttgart .de
538
0-8186-5530-5194 $3.00 Q 1994 IEEE
2 Basic temporal frameworks
To systematically develop a complete set of temporal in-
terrelations for multimedia, we examine how many and
which relations are theoretically possible. Depending on
their elementary units, two basic classes of temporal mod-
els can be distinguished [vBee92], [Rich89]. In the first
class, time is expressed by means of points in a one-dimen-
sional time space [ViKa86] whereas, in a second model
class, intervals are the atomic units of the time space
[Alle83]. This section introduces the basic temporal frame-
works, their elementary units and the relations between
them.
539
[LiKo92], [RoDe92]. So, we do not need to specify the resentable within the model. Since it is not possible to enu-
temporal behavior at exactly one point in time rather it is merate all potential scenarios, we characterize the expres-
sufficient to specify the temporal behavior close to each sive power for a number commonly used temporal models
point in time. This implies that there is not any perceptible by the temporal relations that are expressible. Our analysis
difference in the presentation if somebody specifies for two determines the PRs and IRs of the selected temporal mod-
events el and e2 that el < e;! or in the second case el s 9. els and gives a classification whether it is mainly point- or
This holds because the audience cannot distinguish wheth- interval-based.The latter question is not always easy to an-
er el is simultaneous to e2 or el is 1millisecond before e2. swer because some temporal models use intervals as their
Therefore, it is sufficient to be able to express only one of basic units but their relations address at most one end-point
the relations < or 5 . In this paper, we operate with the rela- of each interval. However, those models have essentially
tions < and >, and do not need the relations s and 2. Ana- the same characteristics as point-based approaches.
logically, the relation * differs from the ?-relation only in
one point in time. Since there is not perceptible difference
between the two relations, we do not need the relation if
4.1 Time-line
animation
we have the relation ?. Observe that we need the relation ?
if any basic PR can hold between two events. This indefi- time
nite often occurs during the specification and planning pro- b
cess when not all events are known yet. FIGURE 1. Tlme llne model
Finally, the PR 0 might be used to spe cify inconsisten-
cies because 0 means that no temporal relation between The time line model is applied by [BHL91], [Gibbgl],
2 events exist. Consequentially, only either of the events [Applgl], [Drap93] and in HyTime [HyTi92]. In the time
can occur. In this paper, we focus on consistent scenarios. line model, all events are aligned on a time axis (time-line)
So, we do not consider the inconsistent relation 0. as it is shown in Figure 1.Since events are the atomic units,
To summarize, the relations e, = , > and ? are the rele- the time line model is point-based. All events are totally or-
vant PRs in multimedia environments. Powerful point- dered on a time line. So, exactly one of the PRs <, =, >
based temporal models should be able to express at least holds between any pair of events on a single time line. As
this set of relations. all events are totally ordered, it is impossible not to define
According to Section 2.3, the PRs <, =, > and ? generate a relation between any two events. This means that the re-
the 29 interval relations. An enumeration of the 29 IRs is lation ‘?’ cannot be expressed in the time-line model. This
given in [WaRo93]. Since at most 187 (Table 3) out of 2131 lack of flexibility is a major disadvantage of the time-line
= 8191 consistent IRs have a point-based conjunctive nota- model. With e, = and > being the only possible PRs in the
tion, we qualified only 187-29 = 158 as irrelevant for mul- time-line model, we can conclude that the 13 basic IRs are
timedia by looking at the PR notation. So, what is the rele- the only IRs that are expressible in the time-line model.
vance of the 8191-187 = 8004 indefinite IRs that cannot be
represented by a PR conjunction? We found that not all of 4.2 Temporal point nets
them are irrelevant. One of the relevant IRs is the relation
5
simultaneous
‘not parallel’ which is needed when limited resources are
shared. For example, if there is only one loudspeaker, then animation
two incompatible audio sequences should not be presented simu taneo
simultaneously.Therefore, we specify the audio sequences vi& audio
not to be parallel. This is expressed by the indefinite IR {<, e ore
m, mi, >}. Represented by PRs, a disjunction is needed:
FIGURE 2. Temporal polnt nM
Ex s By v Ey s Bx. Consequently, ‘not parallel’ cannot be
represented in point-based systems that do not allow dis- [BuZe93] use a point net to represent time specifications
junctions. (Figure 2). Relations address events establishing temporal
equalities (=) and temporal inequalities (<, >). Although
4 Evaluation of multimedia time models [BuZe93] does not mention it, a fourth relation (?) can be
specified meaning: The relation between two time points is
To specify the temporal interrelations of media items, not restricted. The ?-relation adds a flexibility to the model
several representations techniques based an different tem- that cannot be found in the time-line model. Using the PRs
poral models have been proposed by various authors. Each <, =, > and ?, 29 IRs can be represented including the 13ba-
technique uses abstractions about time. When choosing a sic IRs.
temporal model, one of the criteria is its expressive power, [BuZe93] also defines a relation construct ‘before by at
which is defined by the multimedia scenarios that are rep-
540
least 6' where 6 is a delay parameter describing the tempo-
ral distance between two events. This is why, the point re-
lations 5 and z can be specified as well. In total, the PRs c, sequence parallel-first parallel-last
s,=, 2 , >, ? are representable in the point net model gener-
FIGURE 4. Path expressions
ating 82 IRs.
single PR. The sequence operator models a relation be-
tween the end-point of the first and the beginning of the
4.3 Timed petri-nets second interval. The IRs that can be expressed by the se-
h
@+@-+o
end-end
3+ parallel-last are identical because the attributes first and
last give reference points for subsequent operators, which
do not have any impact on our relation analysis. The paral-
!z
end-begin
lel operators establish a relation between the start-points of
two intervals. 3 indefinite IRs are expressible by the paral-
=: begin-end lel operators: {s, =, si}, {di, 0,fi, m, <} and {>, mi, oi, f, d}.
. - To summarize, path expressions are only able to represent
FIGURE 3. Petri nets 7 IRs: 4 basic IRs {m}, {mi}, { c } , {>} and 3 non-basic in-
A timed petri net model is proposed by [LiGh90] and definite IRs {s, =, si}, {di, 0,fi, m, c } , {>, mi, oi, f, d}.
[Hoep91]. The petri net of [Hoep91] is a mapping of the MHEG (Multimedia Hypermedia Expert Group)
path notation on petri nets and will be analyzed together [MHEG92], [KrCa92] is a standardization group to estab-
with the path notation in Section 4.4. In this section, we es- lish a new standard for multimedia objects. MHEG uses
sentially follow the petri net definition of [LiGhgO]. There, two temporal operators sequential and parallel similarly to
intervals are represented by places and relations by transi- the path expression model. We will show in section 5 that
tions. In order to avoid ambiguities, we need the additional the both operators are elements of a more general interval
assumption that petri nets in this context are conflict-free. operator set.
The basic units of the model are intervals. Therefore, this
model is classified as interval-based although transitions 4.5 Summary of the evaluation
refer only to end-points of intervals. Table 4 summarizes the multimedia time models, their
The relation '?' is specified if two places are not con- basic types and the corresponding IRs that can be repre-
nected by any transition. As shown in Figure 3, c , =, > can sented. It can be observed that non of the examined tempo-
be modelled by a transition in conjunction with a delay ral models exceeds the expressive power of the point-
place 6. The delay place represents an idle time 6 C !I?+,. If based framework, not even those models that operate on
6 is in !I?+, the corresponding relations are c and >. The re- intervals.
lation = is modelled if 6 = (0). In this case, the place can
be omitted as it is done in Figure 3. If 6 is unrestricted in number of interval relations
%+, then 5 or 2 is expressed. time model type representable
In petri nets, the PRs s,c, =, >, z can be represented. total basic by t h e P y
Since Figure 3 assures that any combination of interval
<, =, >, .
time-line point-based 13 13 13
end-points can be connected by a relation, the petri net
polnt nets point-based 82 13 29
model is as powerful as the point net model. This means
petri nets interval-based 82 13 29
that 82 IRs can be expressed although [LiGh90] described
path expressions interval-based 7 4 7
only the 13 basic IRs.
TABLE 4. Summary: Multimedia time models
541
duration. Also, the interval-based framework covers more
relations than the point-based. Therefore, a complete set of
interval-based abstractions for multimedia is developed in
this section.
542
The 10 interval operators are a complete set to represent ed, interval operators or a subset such as the path expres-
the 29 relations generated by <, =, > and ?. But this does not sions are applicable.With the emerging interactive media, it
imply that all operators are needed to define a complete is expected that the advanced models are used more fre-
model for a multimedia environment? Sometimes, only a quently.
selection of the operators is necessary. E.g., if the duration
of all media items is preknown and fixed, the temporal
References
[Me831 J. F. Allen. Maintaining Knowledge about Temporal Intervals.
model may be restricted to the operators taking at most 1 Comm. ACM, 26(11):832843,11 1983.
delay parameter. Note that the requirements ‘preknown and [Appl91] Computer Inc. Apple. QuickTime Developer’s Guide.
Developer technical publications, 1991.
fixed duration’ are very strict and prohibit any kind of in-
[BDF+92] Ingo Barth, Gabriel Dermler, Franz Fabian, Kurt Rothermel,
teraction or flexibility. With the emerging interactive mul- Frank Sembach, Robert Erfle, and Johannes Rueckert. Multimedia
timedia systems, it is expected that a larger subset of the in- -
Document Handling A Survey of Concepts and Methods. Technical
report, IPVR - IBM, 9 1992.
terval operators is needed because interactive media items [BHL91] G. Blakowski, J. Huebel, and U. Langrehr. Tools for
introduce a huge number of unpredictable durations. Specifying and Executing Synchronised Multimedia Presentations.
2nd. Intl. Workshop on Network and Operating System Support for
Digital Audio and Video, Heidelberg, 11 1991.
[Bruc72] B.C. Bruce. A Model for Temporal Reference and its
5.3 Example Applicants in a Question Answering Program.Artificial Intelligence,
3:l-12.1972.
k s l y A 3 r ; 2 .... [BuZe93] M. Cecilia Buchanan and Polle T. Zellweger. Automatic
Temporal Layout Mechanisms. In 1st ACM Intl. Conference on
Multimedia, pages 341 - 350,8 1993.
talk1 talk2 [CaHa74] R. H. Campbell and A. N. Habermann. The Specification of
Process Synchronisation by Path Expressions. In G. &os and
slide x while(3 sec,*) talk x, x €{l,..,n}; J. Hartmanis, editors, Lecture notes in Computer Science No. 16,
slide x before(0) slide x+l, x €{l,..,n-l};
Operating Systems, pages 89-102. Springer Verlag, 1974.
[Drap93] George D. Drapcau. Synchronization in the MAEstro
Multimedia Authoring Environment. In 1st Intl. ACM Conference on
FIGURE 6. Interactive slide show scenario -
Multimedia. aaees 331 340. 8 1993.
[Gibb91] Simbn Cibbs. Composite Multimedia and Active Objects.
A multimedia presentation scenario is used to demon- Proc. OOPSZA’91, pages 97-112, 1991.
strate the interval operators work. Figure 6 shows how an [Hoep91] P. Hoepner. Synchronisation der Praesentation von
Multimediaobjekten-Modellund Beispiele. In J. Encarnacao, editor,
author specifies a slide show applying interval operators. Informatik-Fachberichte 293, Telekommunikation und multimediale
Any slide and its corresponding talk are presented simulta- Anwendungen der Informatik, pages 455-464. GI, Springer-Verlag,
10 1991.
neously. Before each talk, there is a silence of 3 seconds to [HyTi92] HyTime. Information technology - Hypermediaime-based
give the spectator a first impression of the slide. After a talk Structuring Language (HyTime). ISOIIEC DIS 10744,8 1992.
[KeLo91] Somnuk Keretho and Rasiah Loganantharaj. Qualitative and
is finished, the user can look at the slide silently or continue Quantitative Time Interval Constraint Networks. Proc. of ACM, San
to the next slide interactively. Antonio, 1991.
[KrCa92] Francis Kretz and Francoise Calaitis. Standardizing
In contrast, this scenario is not representable in the time- HvDermedia Information Obiects. IEEE Communications Magazine,
line model because the end-points of the slides is deter- 5 i‘992.
[LiGh90] T. D. C. Little and A. Ghafoor. Synchronisation and Storage
mined interactively. This means that the end-points are not Models for Multimedia Objects. IEEE Journal on Selected Areas in
known ahead of time. However, we need the end-point of Communications, 8(3):413-427,3 1990.
[LiKo92] T. D. C. Little and F. Koa. An Intermedia Skew Control
the previous slide to specify the beginning of the next slide. System for Multimedia Data Presentation. In 3rd Intl. Workshop on
We would have to pick a point on the time-line although we Network and Operating Systems Support for DigitalAudio and Video,
pages 121 - 132,11 1992.
do not know when this point in time will be. This specifica- IMHEG921 ISO/IEC/WD MHEG. Information Technology - Coded
L ~~
tion problem of the time-line model is caused by its lack of Represkntationof Multimedia and Hypermedia Informaiion Objects.
Working Draft 5 , ISO/IEC, 3 1992.
flexibility, i.e. the time-line requires a total specification of [Rich891 Michael M. Richter. Prinzipien der Kuenstlichen Intelligenz.
all temporal relations between media items not admitting B.GTeubner Stuttgart, 1989.
[RoDe92] K. Rothermel and G. Dermler. Synchronization in Joint-
any indeterminism. Consequently, the time-line model is Viewing Environments. In 3rd Intl. Workshop on Network and
not appropriate for partial specifications or interactive me- Operating Systems Support for Digital Audio and Video, 1992.
[Stei90] Ralf Steinmetz. Synchronisation Properties in Multimedia
dia environments. Systems. IEEE Journal on Selected Areas in Communications,
8(3):401-412,4 1990.
6 Conclusion [Stei92] Ralf Steinmetz. Multimedia Synchronisation Techniques:
Experiences Based on Different System Structures. In Multimedia
Communications, Monterey, USA CA, 4 1992. 4th IEEE COMSOC
The very intuitive and natural time-line model is suffi- International Workshop.
cient for specifying deterministic scenarios. However as [vBee92] Peter van Beek. Reasoning about qualitative temporal
information.Artificial Intelligence, 8(956):297 - 326, 12 1992.
soon as interactive media are added, indeterministic tempo- [ViKa86] M. Wain and H.A. Kautz. Constraint propagation algorithms
ral relations are present requiring more expressiveness. Ad- for temporal reasoning. In AAAI-86 Philadelphia, PA, pages 132 -
144,1986.
vanced point-based models might be applied such as the [WaRo93] Thomas Wahl and Kurt Rothermel. Representing Time in
petri net or the point net. In cases when the expressive pow- Multimedia Systems. Technical Report 12, Universitaet Stuttgart, 10
1993.
er is still insufficient or higher-level abstractions are need-
543