A multi-agent system integrating a learning process

applied to Financial market

Férial BENACHOUR HAIT

Université Paris XII. Département Maths-informatique.
61, avenue du GAL de Gaulle. 94010 Creteil Cedex
benachour@univ-paris12.fr

Abstract

This paper presents a Multi-agent System (MAS) applied model to design a financial
market system. The key impact of an agent approach is that the behavior of the
whole system can be understood by fragmenting it into many small autonomous and
interacting units and by specifying the behavior of each unit. The agents actions in
their environment are inductive. It means the market agents are allowed to adapt and
learn from their environment by recording the success and the failure of their past
actions. It is shown in the paper how agents make or loose money according to their
ability for learning from their environment. An interesting result concerns agent
oriented simulation in social systems where various issues can be investigated to
study the emergent dynamic of the system and understand more about the
relationship between micro-levels specification of individual entities and the macro
level emergent behavior of the whole system. Our experimental results show clearly
emergent features like the mass effect and the correlation appearing between the
number of agents and their enrichment's speed and performance in the market.

Key-Words: Objets, Agents, Learning, Financial market, Ada’95 .

1 Introduction

Financial markets become every day more and more complex because various
and uncontrolled parameters induce the market behavior. Traditional economic
models are built upon the rational expectation (RE) paradigm [GRO 81] which holds
three important assumptions :
1. The complete information: all agents are assumed to have full knowledge of the
problem.
2. The perfect rationality: all agents are assumed to be perfectly able to deduce
their optimum behavior no matter how complex is the problem.
3. Common expectations: all agents are assumed to know that all others are
working with the same information on the same "perfectly rational" basis. And
they know that the others know this too.
However RE theory in practice runs into a number of difficulties especially because
the three conditions listed above can rarely be together satisfied and contrast deeply
with our intuition of the way markets work in practice. Towards that many

1
researchers like Arthur [ART 91], Palmer and Holland [PAL 94] studied the RE
paradigm and discussed it deficiency in complex problems. (1) In the case of
complete information : agents may have to learn about the context when the game is
being played out. Problem contexts may them selves not be fully defined initially,
only becoming explicit through the choices of the agents. (2) For the perfect
rationality: agents are not clever enough to compute their optimum behavior, and
even if they have the power they prefer using rules of thumb that have worked
elsewhere. (3) For the common expectations: different agents may well have
different information about a situation, and they cannot rely on others to duplicate
their own reasoning.
More over economic analysis has largely avoided questions about the way in which
economic agents make choices when confronted by perpetually novel and evolving
world and how agents learn from their environment.
All these difficulties added to empirical regularities that can not be explain by
theoretical models, are an indication that standard tools and formal models are not
appropriate for handling this kind of systems and that an other kind of models
should be used to understand financial markets.
The construction of such maintainable, extensible and reliable systems requires the
use of modeling techniques that support abstraction, structuring, modularity and
inheritance.
Among the most recent methods used in software engineering, object oriented (OO)
design methodology [RUM 91] and object oriented programming [SMI 96] have
become important concepts in the design of complex systems. Object oriented
approaches are based upon the central notion of object which encapsulates state
information as a collection of data values and provide behaviors via well-defined
interfaces for operations upon that information [KIN 96]. "Because (OO)
programming models both the behavioral and informational complexity of a system,
the system tends to be much better organized than it was simply well structured.
Because systems are better organized, they are easier to understand, debug,
maintain, and evolve."[IRV 97]. Likewise agent paradigm is also based on the
notion of autonomous and reactive entities embedded in changing uncertain worlds.
If agent is not directly an object, something becomes an agent by the fact that one
has decided to control and analyze it's behavior in mental terms, e.g., beliefs and
goals [UHR 97].
Recent and advanced research [JEN 98] allow us each year to more understanding
and defining the notion of agent. And as D.Kinny pointed out : "we can easily
imagine an extension of the object paradigm to the agent by building upon and
adapting well understood techniques of OO design and programming".
Our present work is rooted in this approach and propose the use of multi-agent
systems (MAS) for the study of financial market behavior. Therefore we consider
the market behavior as a distributed market agents who deal autonomously with
local task planning. The behavior of the simulating environment as a whole is then
an emerging functionality of the individual skills of, and the interactions among the
agents [Tan 98].
The agents action in their environment are inductive. It means the agents are allowed
to adapt and learn from their environment by recording the success and the failure of
their past actions. To realize that, we implemented a process of reinforcement
learning with a credit assignment in the agent's internal state. It is shown in the paper
how each agent learns from his experience and updates his own data after each

2
period. As a result we conclude on the efficiency of learning from the environment
and how agents gained performance in term of wealth through the time. Our model
does not necessary converge to an equilibrium and can show long periods of calm
followed by sudden burst or volatility when the agents change some of their internal
characteristics like risk aversion or wealth or learning. The paper road map :
Section 2 describes the markets, the participants, the rule of the game and the market
stock price computing. Section 3 introduces the modeling process and the
conceptual model design. Section 4 investigates the implementation issues and
discusses the results. The last section leads to the conclusion and future works.

2 The artificial stock market

Using the inductive approach [PAL 94], we consider the market behavior exactly
at the opposite of the RE theory with agents having little knowledge or reasoning
ability. Agent's activity is involved by his action which is provided by an internal
decision. A credit assignment is affected to it's performance (increasing, decreasing)
through a process of reinforcement learning that compute the feedback returned by
the environment. The agents trade on the basis of internal models or rules that are
initially very poor and perform their actions by observing the success or failure of
their past transactions.

• The participants: To start the game's market, we consider a number of simulated

investors whom buy and sell with a trader. A board data price is there to display
the current prices and transactions made by the agents. Agents are risk neutral
and may be homogenous or not in their interactions.

• Rules of the game and the market specification:

We consider a financial market where a share is exchanged [BIA 97]. The
process follows the next protocol :
1. The display of the public price for all agents.
2. Each investor, if he can propose an order which is a bid to buy or an offer to sell
to the trader for the stock shared on the market.
3. The trader revises his evaluation on the value of the share he holds.
4. The trader gives his new quotations and displays the price transactions to the
investors and the board price.
5. A new public price which is the average of the best prices of offers and bids is
displayed on the board price

• The market and Stock Prices

1. When do the investors buy or sell? The fundamental value of the stock at any
trading period is given by D/R "(the dividend divided by the interest rate; this
assumes that the trader believes that the current price is the best estimator of the
future price, and the dividend will remain constant. That is a price higher than
D/R should make it profitable to sell the stock and hold cash to earn interest ,
and a price lower than D/R should make it profitable to hold the stock"[TAY
95]. In this sense, D/R is the fundamental value of the stock ; since the interest

3
 rate is constant at 10% in this simulated stock market. The fundamental
price/dividend ratio should therefore always be 10.
2. How are the orders computed? The order Q(t) at time t for each investor is
computed assuming a maximization of his utility function [Bia 97] upon his
corresponding wealth W and risk aversion ra. This leads to the following
formula :
Q (t ) = Max W [U (t )]

3. How is the price market computed?

Depending upon the prompting of their trading decision rules. Bids and offers
are made at the current ruling market price. The market price is than adjusted for the
next trading period t+1 according to:

Price t+1 = Price t *[1+K*(sum of bids -sum of offers)]

where K is a constant. In the next period t+1 the updated price is used, and investors
will seek to trade at this new level.

3 The modeling process

Object-oriented modeling of social systems requires the integration of new

concepts related to mental process such as learning, planning and communication
[KUH 97]. For a complete specification of the system, we need to consider our
model at two levels of abstraction:

a) From the external point of view : The system is decomposed in modeled and
complex agents characterized by their actions, their perceptions and
interactions. This is the analysis phase which leads to the object model.
b) From the internal point of view : All the elements required by each agent
according to his architecture. It means all the different components that underlie
his organization and make him acting and learning in his environment. This is
the agent structure or architecture.

• The process of learning: In its most general meaning, the term "adaptation"
denotes all changes of a system so that it becomes suitable for a given situation
or purpose. In doing so, no explicit distinction is made between adaptation and
learning in the MAS; instead it is assumed that "adaptation" is covered by
"learning". This is in accordance with common usage, according to which the
term "adaptation" is only applied to those self-modifications that enable system
to survive in a changed environment. This meaning however is too broad to be
of value from the view point of the Machine Learning [WEI 96].
For the experiments we applied the principles of the learning feedback that is
available to a learning entity and that indicates the performance level achieved
so far. This criterion leads us to the notion of reinforcement learning that we
used and inspired by some many results of Q-learning [WAT 92].

4
Some important definitions giving the learning parameters:

In the reinforcement learning paradigm, a learning agent continually

receives sensory inputs from the environment, selects and performs actions to affect
the environment, and after each action, receives from the environment a scalar signal
called reinforcement. The reinforcement can be positive (reward), negative
(punishment), or null. The objective of learning is to construct an optimal action
selection policy that maximizes the agent's performance [LIN 92].

Def.1: the K-th agent has a constant risk aversion ra which gives a negative utility
function.
U k (W k (t )) = − e − ra K W K (t )

Def.2: we assume that an indicator of estimated state for an agent Ak at time t is the
evaluation of his action utility during this period.

E [A K (t )] = r (t ) + G ∗ U K (W K (t ))
where G is the reinforcement received and r is the yield of wealth obtained after
each transition from t-1 to t .

Def.3: a natural measure of an agent performance V(t) is the measure of his

cumulative past experiences upon all his estimated states from t=0 to t.
t
V (t ) = ∑ E ( A K ( j ))
j=o

The learning protocol: for short the agent at time t learns as follows

1. Receives the new price from the environment

2. Computes his yield r and his credit assignment G
3. Records r
4. Estimates his state and computes E
5. Computes and evaluates his action (the order + (buy/sell))
6. Computes his performance V
7. If V(t)>0 than executes his action; else non
8. t ← t + 1 ; returns to 1 .
The process of learning and the designed model are detailed and described in our
previous works (see [BEN99 a,b] ).

• The Analysis
Using a design methodology based on a simplified version of Fusion designed
by Coelman and al [COE 96], the above specification of the market in section 2, can
be analyzed and a design created for an implementation. For that it is appropriate to
identify the objects and system actions from the written specification of the market
structure. The nouns become the objects and major verbs become the system actions.

5
From Fusion we used the analysis methodology to provide the OMT model of
agents or the object model.
The object model : represents the whole system and shows the interaction between
the objects that constitute the system. Using all the objects identified, the artificial
market can be represented by the OMT model of agents (see figure 1). The
instantiating of the object model leads to a MAS involving three kind of agents : the
dumb investor (are not allowed to learn), the smart investor (are allowed to learn),
the trader and a board price.

Smart Investor Dumb investor

Board Price
Displays Internal state Internal State

Prices

Trader
Buys Sells

Transactions

Gives

Figure 1. The OMT model of agents

The agent architecture:

A formal and abstract view of agents is that an agent consists of an object
with a set of properties (values, data) and a set of methods (actions, functions,
operations) [FER 95]. Since objects are the basic run-time entities of our system
modeling, each object will be represented by an encapsulated unit or module.
Modularity becomes one of the main characteristics and advantage of OO paradigm.
This enable us to consider objects as concurrent actors that communicate
asynchronously [LOE 97]. We suggest an investor architecture defined by the
following 5-tuple (see figure 2). The environment holds the other agents and the
board price.
U: is the set of data received by the agent from his environment like the current
stock price, it's dividend, it's ratio or the transaction price sent by the trader.
KB: is his knowledge base according to his individual parameters like the wealth
and the risk aversion.
A: is the set of actions or transactions performed by the agent at each period in his
environment.
M: is the memory space of the agent. It records for every period all its variables
(instances).

6
E: is the evaluation process. It updates systematically all the agent's characteristics,
updates his variables for each period and evaluates it's performance through the
process of learning with a credit assignment that computes all the previous state of
the agent.
The perception and action sets design the external state of an agent interfacing with
it's environment. It's the communication module.

The memory space and the evaluation process design the internal state of an agent
that make him performing his actions to achieve his goal. It's the evaluation module.

Environment

Communication External
U Module A State

KB M Evaluation E Internal
Module State

Data flow

Figure 2. Agent architecture

4 Implementation and results

An agent in our market consists of components for perceiving its

environment (sensors), for keeping an internal state (knowledge base), for
communicating and interfacing with other agents and performing his actions to
achieve his goal. The market it self or the MAS refers to a distributed simulation
environment, where simulation takes place as a sending of messages between these
concurrently interacting entities. Moreover because investors performs their
transactions with the trader while perceiving the current price from the data board or
receiving the transaction price from the trader, our agents system components are
intended to run in parallel.
To fulfill the above requirements, the experimentation requires suitable tools. In
Ada'95 programming language [ISO 95], one can also write programs that perform
more than one activity concurrently (during the same period of time). This
concurrent processing is calling tasking [NAI 95]. In Ada terminology, a process is
represented as a task. Ada's task type mechanism provides support for a modular
design method, in which tasks work on encapsulated local data. The communication
is made by rendezvous. To implement one or more object classes of the system, we

7
use Ada package. A package consists of one or more object classes implemented as
abstract data type [BUR 97].
Since agents are concurrent entities possessing their own logical processor, a task is
attached to each agent as a record component implementing the agent's concurrent
dynamical behavior and then encapsulated in a package class-agent. The board
price is a package displaying and updating the current prices. The trader is a reactive
agent encapsulated in a package class that systematically computes the quotations
for each investor transaction. Real time features like rendezvous are used to realize
communication mechanism between agents.
The fore going has described how the micro-level behavior of the modeled entities is
constructed. All model outputs are the result of their complex interactions. We used
9 agents 8 investors and a trader during 200 or 300 periods. All the investors are not
homogenous and have individual wealth risk aversion and learning ability for their
own transactions.

• Heterogeneity of the market :

we start the description of the results by the study of the dynamic of market
prices when investors acting are either totally smart (CPSA), totally dumb (CPDA)
or mixed (CPMA). Figure 3 shows the evolution of the market price.

3,00E+01
2,50E+01
2,00E+01 CPMA
1,50E+01 CPDA
1,00E+01 CPSA

5,00E+00
0,00E+00
1
35
69
103

137
171
205
239
273

Figure 3. Dynamics of prices for different agents

When the agents are smart or mixed, the variability of the market price is low.
We interpret these results as saying that the market converges to an equilibrium
fairly quickly when agents learn from their experiences. When we increase the dumb
agents we obtain a market price which is more volatile than before. At the beginning
of the market, prices show a quite hesitation until the middle of the period where the
prices become more volatile. This illustrates that the agents cannot adapt their
actions to the environment.

8
• The performance of the dumb and smart agents

Figure 4, shows the variation of wealth of each type of agent: in a mixed market.
The dumb ones (WDAMM) and the smart ones (WSAMM).

3,00E+04
2,50E+04
2,00E+04
1,50E+04 WSAMM
1,00E+04 WDAMM
5,00E+03
0,00E+00
-5,00E+03 1 45 89 133 177 221 265 309 353

Figure 4. Variation of wealth in a mixed market

The speed of learning is clearly shown in the plot. Agents having an ability for
learning gain in performance and become much richer quickly. The dumb ones can
hardly follows the others because they act with out any reasoning. In fact they
rapidly crash in the market.

• The speed of performance in a homogenous market

Figure 5, shows the speed of agent's performance gaining wealth when acting in a
homogenous market. We can see that the dumb agents in a dumb market (WDADM)
have more chances than before to maintain their actions in the market and raise their
wealth even quite slowly. However the smart agents in a smart market (WSASM)
become less rapidly richer than they were in the mixed market because they are
confronted to other agents concurrency.

5,00E+03
4,00E+03
3,00E+03
WSASM
2,00E+03
WDADM
1,00E+03
0,00E+00
-1,00E+03 1 21 41 61 81 101

Figure 5. Performance of agents in a homogenous market

9
This last result seems very interesting because it's very close-fitting to the reality. In
fact in any market many irrational agents make transactions for a simple reason of
liquidity, and it's very hard for the "smart" agents to maintain a coherent trend of
learning in this kind of environment. However it seems much easier to increase our
richness in a more homogenous market were the agents follow roughly the same
behavior.

• Some emergent features

Figure 6 shows how the price is gaining in volatility when the number of agents is
gradually increasing from 3 agents (Res3) to (Res9). In fact more the number grows,
more the market is dynamic and the prices evolving. This shows an interesting
feature explaining the market efficiency and the evolution trend of the market
through the number of agents acting together.

1,00E+03 Res9
9,00E+02
8,01E+02 Res8
7,01E+02
6,01E+02 Res7
5,02E+02
4,02E+02 Res6
3,02E+02
2,03E+02 Res5
1,03E+02 Res4
3,36E+00
-9,63E+01 Res3
1 17 33 49 65 81 97

Figure 6. Prices variation with increasing number of agents

Figures 7 and 8 shows what we called the mass effect in terms of conditions of rapid
enrichment in a market. Here we are not surprised to see that more the number of
agents is increasing in a market, less the agents have chance to become quickly
much richer. Even if they are smart in majority, the speed of performance seems
deeply correlated with the agents weight. Thus even if an agent has an important
wealth when starting the game, the concurrency with other agents plays a non
neglected role in the speed of his enrichment.

10
 4,00E+04

3,00E+04 Rich5d
Rich5c
2,00E+04
Rich5b
1,00E+04 Rich5a

0,00E+00

11
21

31
41
51

61

71
Figure7. agents enrichment in a market with 5 agents

Rich9h
4,00E+06
Rich9g
3,00E+06 Rich9f
Rich9e
2,00E+06
Rich9d
1,00E+06 Rich9c

0,00E+00 Rich9b
1 9 17 25 33 41 49 57 65 73 Rich9a

Figure8. agents enrichment in a market with 9 agents

5 Conclusion and future works:

Our first objective in studying financial markets was to conceive and build
an adapted model to understand some properties of the stock market. In this I
believe we were successful, designing and implementing an artificial stock market
with different type of agents.
Our second interest was to enhance understanding of how OO design might be used
and applied to improve MAS. Here we believe that our results go towards this
approach suggesting that MAS could be suitable tools for understanding complex
mechanisms.
Our third objective was to apply an evolutionary approach like inductive process to

11
provide agents decision in the market. Even if the stock market is much simpler than
the real thing, it still contains considerable complexity; Because investors both
create and exploit the prices series, the agents are essentially coevolving by making
mental models or hypotheses, based on past experiences and training.
We showed in the experimentation part that a market populated by agents acting
according to some simple rules and learning, can generate rich dynamics in market
price. Agents make or loose money according to their ability for learning from their
environment. They adapt gradually their transactions to the market behavior.
Besides that the criterion of homogeneity gave an interesting result concerning the
agents behavior and their impact in the whole market dynamics.
An other important thing to point out with a multi-agent simulation is the possibility
to give quantitative answers to problems of qualitative aspects of social science like
the design of the behavior which is very hard to formalize in theory, and seems more
tangible with this kind of systems where the actions of agents are well designed.
A number of variations to the stock market model are possible, and the work is still
ongoing. Many decision variables like agents characteristics can be changed to
observe the market dynamics through the agents behavior. In this sense we can
actually show that the risk aversion parameter play often an important role in the
agents transactions. That is a relationship between and agent's enrichment and his
risk aversion can be comforted deeply.
The last point concerns agent oriented simulation in social systems where various
issues can be investigated to study the emergent dynamic of the system and
understand more about the relationship between micro-levels specification of
individual entities and the macro level emergent behavior of the whole system. The
many emergent features go towards that and can lead to many others characteristics
Of the studied system.

Acknowledgement

I would like to thank Konstantin Verchinine, Brahim Chaib-Drâa, Suzanne Pinson

and Pr Steve Lai for their suggestions and comments on an earlier draft of this
article.

6 Références

[ART 91] W.B.Arthur, «Designing Economic Agents that Act Like Human
Agents»,
in Learning and adaptive economic behavior, May 1991. pp 353-359.
[BEN 98] F.Benachour Hait, « Towards a formal approach of artificial adaptive
agents to a stock market » , Technical Report : TR- 14, 1998.
[BEN 99a] F.Benachour Hait, « Méthodologie et développement d’agents artificiels
adaptés à un marché financier », Technical Report :TR-19, 1999.
[BEN 99b] F.Benachour Hait, « Une méthode d’apprentissage pour un système
multi-agents: le cas d’un marché financier » (submitted).
[BIA 97] B.BIAIS, Microstructure des marchés financiers, Puf, Paris, 1997.
[BLU 92] L.Blume and D.Easley, « Evolution and Market behavior », in Journal Of
Economic Theory, n°58, pp 9-40, 1992.

12
[BUR 98] A.Burns and A.Wellings. Concurrecy in Ada. Cambridge University
Press, 1998.
[COE 96] D.Coelman et al., Fusion La méthode orientée objet de 2 ème Génération,
0Masson, 1996.
[FER 95] J.Ferber. Les systèmes multi-agents. Interéditions, Paris, 1995.
[GRO 81] S.J.Grossman, «An introduction to the theory of rational expectations
under asymmetric information», Review of economic studies, XLVIII, 1981, pp
541-559.
[HIL 96] D.RC.Hill, Object-Oriented Analysis and Simulation, Addison-Wesley,
1996.
[IRV 97] A.Irving Littlefield III, « An object oriented approach to automated
Generation of Challenge Examination Using Ada’95 », in ACM Ada letters,
VOL XVIII n°1.
[ISO 95] International Standardisation Organisation , « Information Technology-
Programming Languages », Ada International Standard ISO/IEC 8652, 1995.
[JEN 98] N.R.Jennings and M.J.Wooldridge, Agent Technology : Foundations
applications and Markets, M.J. Wooldridge (ed), Springer, 1998.
[KIN 96] D.Kinny, M.Georgeff and A.Rao, « A methodology and modelling
technique for systems of BDI Agents », in MAAMMAW’96, in LNAI , Agents
breaking away, pp 56-71.
[KUH 97] R.Kuhnel, «Agent oriented programming with java» in Proceeding of
the search international conference on artificial intelligence and information
control system of robots, 1997.
[LIN 92] L.J.Lin, «Self-Improving Reacting Agents Based On reinforcement
Learning, Planning and Teaching», in Machine learning ,8, 293-321, 1992.
[LOE 97] H.Loeper and al., « Concurrent Objects in Ada’95 », in Ada Letters VOL
XVII n°6, 1997.
[NAI 95] D.J.Naiditch, Rendezvous with Ada'95, Naiditch edition, 1995.
[PAL 94] R.G.Palmer, W.Brian Arthur, J.H.Holland and al, « Artificial economic
life: a simple model of a stockmarket», in Physica D, n°74, 1994, pp 264-275.
[RUM 91] J.Rumbaugh et al., Object Oriented Modeling and Design, Printice Hall
international, 1991.
[SMI 96] M.A.Smith, Object oriented software in Ada'95 , In. Thomson. Com.
Press, 1996.
[TAN 98] G.S.H.Tan, « Applying intelligent agent technology as the platform for
simulation », in Simulation Symposium, 1998.
[TAY 95] P.Tayler, «Modelling Artificial Stock Markets Using Genetic
Algorithms», in Intelligent systems For Finance and Business, John Wiley and
Sons, 1995.
[UHR 97] A.M. Uhrmacher, « Concepts of object - and agent- oriented
simulation», in Transactions of the SCS International, Vol.14, n°2,pp 59-67.
[WAT 92] C.I.C.H Watkins and P.Dayan, «Q-learning », in Machine Learning,
8, 279-292, 1992.
[WEI 96] G.Weiss, «Adaptation and learning in multi-agent systems: some
remarks and a bibliography, G.Weiss and S.Sen (Eds), pp(1-21), LNAI , Vol
1042, 1996.

13