Learning classifiers

Rodrigo Barbosa de Santis

May 21, 2019

1 Introduction
Although the first methods developed in Machine Learning area presented great performance predicting vari-
ables in continuous space, the same efficiency were not meet when learning from discrete outputs – such as Yes/No,
’Boy’ or ’Girl’, or ’Cold’/’Warm’/’Hot’ – due to its non-linearity.
Several methods were proposed since then so learning machines could be applied in these cases, which have
became known after as classification problems. In the current study, it is applied adaptive linear element (Adaline)
and the McCulloch-Pitts pattern recognition model using Rosenblatt training rule.

2 Materials and methods

2.1 Adaptive linear element – Adaline
The adaptive linear element, or Adaline, is a linear classifier which applies a threshold function after fitting a
linear regression through least mean square (LMS) algorithm. Figure ?? presents a schematic model for Adaline
training.

Figure 1: Adaline training model. (Principe et al., 1999)

Classes are defined according to the linear weight output: if greater than a value set, the element is classified
as ’A’, otherwise it is classified as ’B’. In the example, the elements with weight greater than 0 are considered
’healthy’ and lower as ’sick’.

2.2 McCulloch-Pitts
McCulloch-Pitts processing element (PE) is a sum-of-products followed by a threshold nonlinearity, represented
in Figure ??, mathematically described by (Principe et al., 1999):
(D )
∑
y = f (net) = f w i xi + b (1)
i=1

where xi and yi are the network input/output, D is the number of inputs, wi are the weights and b is a bias. The
activation function f (net) is the threshold function given by:
{
1 for net ≥ 0
f (net) = . (2)
−1 for net < 0

1
 Figure 2: A two-input, one output McCulloch-Pitts PE. (Principe et al., 1999)

Rosenblatt proposed a procedure to minimize the error by changing the weights of MCP, described by the
following equation: (Principe et al., 1999)

w(n + 1) = w(n) + η(d(n) − y(n))x(n) (3)

where w is the weights, η the step size, y the net output and d the desired response.

2.3 Dataset
Three datasets were used in this work: two uni-variate data and one bi-variate dataset, shown in Table 1. Three
separate .csv (common separated) files were created for each dataset.

Table 1: Datasets used

Table 3: data3.csv
Table 2: data2.csv
x d Table 4: data4.csv
x d 67 0
67 0 88 0 x1 x2 d
88 0 111 0 -0.50 0.35 0
111 0 130 0 -0.75 0.85 0
130 0 134 0 -0.60 0.65 0
134 0 175 1 -0.50 0.75 0
208 1 207 1 0.50 0.00 1
235 1 280 1 -0.30 -0.20 1
240 1 330 1 0.20 0.10 1
280 1 385 1 0.10 -0.10 1
296 1 421 1
551 1

3 Development
The algorithm is implemented using Python 2.7.8 (Van Rossum, 1998), including the following libraries:

1. NumPy (http://www.numpy.org/) – A large set of funcations that allows arrays manipulation;

2. PyLab (https://pypi.python.org/pypi/pylab/) – A scientific library, which provides a group of graphic and
chart functions;
3. Scikit-learn (http://scikit-learn.org/) – Machine learning library in Python.

The parameters set for the method are summarized in Table 5. All weights w were initialized as 0.00 and the
threshold point as 0.5.

Table 5: Parameters set for methods training

data2 data3 data4

Learning rate 10-5 10-5 10-5
Epochs 100 100 100

2
4 Results
The results for uni-variate datasets – data2 and data3 – are shown in Figure 3. Adaline correctly classified the
first case, where data were balanced (5 of each class). In the second problem, where there were more elements of
the kind ’B’ than the other class, the fitted line was unbalanced. Therefore, one of the elements were classified as
type ’A’ by the model, although it was a type ’B’ element.
MCP classified all data correctly in most cases, although in several the method did not converge to a proper
result, as shown in its application in Problem 2, where the threshold found was over all classes.

Figure 3: Uni-variate datasets results

In the last problem, Adaline achieve an accuracy of 100%, developing a threshold surface as shown in Figure
4. Perceptron also achieve a 100% accuracy for the problem.

5 Conclusion
Adaline was an important step for extending the regression method for the classification problem. Although, it
presents some limitations when analyzing unbalanced datasets, which may harm its application in some cases.
McCulloch-Pitts approach brought the output error for inside the training, and the Rosenblatt training presented
a regular procedure for net training based on randomness. It may or not achieve a feasible solution, so a single
execution is maybe not recommended.

References
Principe, J. C., Euliano, N. R., & Lefebvre, W. C. (1999). Neural and adaptive systems: fundamentals through
simulations with CD-ROM. John Wiley & Sons, Inc..

Van Rossum, G. (1998). Python: a computer language. Version 2.7.8. Amsterdam, Stichting Mathematisch Cen-
trum. (http://www.python.org).

3
Figure 4: Bi-variate dataset results