Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Communalities Factor Analysis and Its Extensions

Karl G J oreskog Slide 1 Uppsala University Slide 3 Many papers on factor analysis at that time focused on the question: What numbers should be put in the diagonal of R to make this approximately equal to , where is a p k matrix of factor loadings? Rc (1)

Alternative Titles Factor Analysis at 100: The Last 50 Years Factor Analysis: 50 Years in 50 minutes

The numbers in the diagonal of Rc are called communalities. 2 Guttman (1956) showed that the squared multiple correlation Ri in the regression of the ithe variable on all the other variables is a lower bound for the communality of the ith variable:
2 c2 i Ri

(2) % $

& '

% $

& '

Uppsala Symposium 1953 Uniqueness

My history of factor analysis begins just about 50 years ago, or 1953, at the Uppsala Symposium on Psychological Factor Analysis. I was in high school then fully unaware of what was going on in Uppsala and I had no idea what factor analysis was. This symposium was hosted by Herman Wold, professor and chair of statistics at Uppsala University. Wold had met with Louis and Thelma Thurstone in Stockholm and he was inspired by their work on factor analysis. There were some prominent people in this symposium including Maurice Bartlett, D.N.Lawley, Georg Rash, and Peter Whittle. The Uppsala Symposium is of minor importance in the history of factor analysis but it had a great consequence for me for six years later Wold suggested that I do a dissertation on factor analysis. & % The counterpart of the communality c2 i is the uniqueness 2 = 1 c . Hence, (2) is equivalent to u2 i i Slide 4
2 ii u2 i 1 Ri = 1/r

Slide 2

(3)

In my dissertation I therefore suggested that

ii u2 i = /r ,

(4)

where is a parameter to be estimated.

&

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Heywood Cases Statistical Formultion

Much of the discussion in the 50s were procedures for choosing communalities and estimating factor loadings. There was a need for a statistical formulation. So in my dissertation, I suggested that one could estimate the covariance matrix subject to the constraint = + (diag 1 )1 (5) J oreskog (1967) solved this problem by focusing on the concentrated t function (8) f () = min F (, ) , 2 which could be minimized numerically. If one or more of the i gets close to zero, this procedure becomes unstable. J oreskog (1977) therefore developed this procedure further by reparameterizing
2 , i = + (ei ) i = ln i

Slide 5

Slide 7

and I investigated a simple non-iterative procedure for estimating and from the sample covariance matrix S. Later I developed a maximum likelihood method for estimating model (5). & ' % $ & '

(9)

This leads to a very fast and ecient algorithm, the use of which has been very successful. % $

Maximum Likelihood Factor Analysis Other Fit Functions

Factor analysis as a statistical method was formulated already by Lawley (1940), see also Lawley & Maxwell (1963, 1971) and it was further developed by Anderson & Rubin (1956). However, as late as the mid 60s there was no good way for computing the estimates. Using well established notation, the problem is to minimize the function FML (, ) = log + tr(S1 ) log S p , where = + 2 , and 2 is the diagonal matrix of error variances. & % & % (7) (6) Unweighted Least Squares(ULS) 1 tr[(S )2 ] 2 Generalized Least Squares(GLS) FU LS (, ) = 1 tr[(I S1 )2 ] 2

Slide 6

Slide 8

(10)

FGLS (, ) =

(11)

Each of these t functions can also be minimized by minimizing the corresponding concentrated t function (8).

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Weighted Least Squares

As more knowledge is gained about the nature of social and psychological measurements, however, exploratory factor analysis may not be a useful tool and may even become a hindrance. (12) Slide 11 Most studies are to some extent both exploratory and conrmatory since they involve some variables of known and other variables of unknown composition. The former should be chosen with great care in order that as much information as possible about the latter may be extracted. It is highly desirable that a hypothesis which has been suggested by mainly exploratory procedures should subsequently be conrmed, or disproved, by obtaining new data and subjecting these to more rigorous statistical techniques.

FV (, ) = Slide 9

1 tr[(S )V]2 2

ULS : V = I GLS : V = S ML : V =
1

(13) (14) (15)

ML = Iteratively Reweighted Least Squares

& '

% $

& ' The basic idea of factor analysis is the following. For a given set of response variables x1 , . . . , xp one wants to nd a set of underlying latent factors 1 , . . . , k , fewer in number than the observed variables. These latent factors are supposed to account for the intercorrelations of the response variables in the sense that when the factors are partialed out from the observed variables, there should no longer remain any correlations between these. If both the observed response variables and the latent factors are measured in deviations from the mean, this leads to the model: xi = i1 1 + i2 2 + + in k + i , (16)

% $

Exploratory Factor Analysis

Exploratory factor analysis is a technique often used to detect and assess latent sources of variation and covariation in observed measurements. It is widely recognized that exploratory factor analysis can be quite useful in the early stages of experimentation or test development. Thurstones (1938) primary mental abilities, Frenchs (1951) factors in aptitude and achievement tests and Guilfords (1956) structure of intelligence are good examples of this. The results of an exploratory factor analysis may have heuristic and suggestive value and may generate hypotheses which are capable of more objective testing by other multivariate methods. & %

Slide 10

Slide 12

where i , the unique part of xi , is assumed to be uncorrelated with 1 , 2 , . . . , k and with j for j = i. In matrix notation (16) is x = + & (17) %

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Two-Stage Least-Squares
The unique part i consists of two components: a specic factor si and a pure random measurement error ei . These are indistinguishable, unless the measurements xi are designed in such a way that they can be separately identied (panel designs and multitrait-multimethod designs). The term i is often called the measurement error in xi even though it is widely recognized that this term may also contain a specic factor as stated above.

y = x+u, Slide 15
1 = S xx sxy , 1 1 1 = (S zx S S zx S zz Szx ) zz szy , 1 1 uu (S zx S , (n p)1 zz Szx )

(19) (20) (21) (22) (23)

Slide 13

Sxx sxy + uu = syy 2

& '

% $

& '

% $

Rotation
= + 2 , where and 2 are the covariance matrices of and , respectively. Slide 14 Let T be an arbitrary non-singular matrix of order k k and let = T Then we have identically

Reference Variables Solution

Partitioning x into two parts x1 (k 1) and x2 (q 1), where q = p k , and similarly into 1 (k 1) and 2 (q 1), (17) can be written Slide 16 x1 x2 = + 1 = 2 + 2 , (24) (25)

(18)

= T

= TT

This shows that at least k2 independent conditions must be imposed on and/or to make these identied. & % &

where 2 (q k ) consists of the last q = p k rows of . The matrix 2 may, but need not, contain a priori specied elements. We say that the model is unrestricted when 2 is entirely unspecied and that the model is restricted when 2 contains a priori specied elements. %

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Solving (24) for and substituting this into (25) gives

Multigroup Analysis
x2 = 2 x1 + u , (26) Consider data from several groups or populations. These may be dierent nations, states, or regions, culturally or socioeconomically dierent groups, groups of individuals selected on the basis of some known selection variables, groups receiving dierent treatments, and control groups, etc. In fact, they may be any set of mutually exclusive groups of individuals that are clearly dened. It is assumed that a number of variables have been measured on a number of individuals from each population. This approach is particularly useful in comparing a number of treatment and control groups regardless of whether individuals have been assigned to the groups randomly or not. & ' % $ where u = 2 2 1 . Each equation in (26) is of the form (19) but it is not a regression equation because u is correlated with x1 , since 1 is correlated with x1 . Slide 17 Let xi = i x1 + ui , (27) Slide 19

be the i-th equation in (26), where i is the i-th row of 2 , and let x(i) (q 1 1) be a vector of the remaining variables in x2 . Then ui is uncorrelated with x(i) so that x(i) can be used as instrumental variables for estimating (27). Provided q k + 1, this can be done for each i = 1, 2, . . . , q . & ' % $

Factorial Invariance Conrmatory Factor Analysis

In a conrmatory factor analysis, the investigator has such knowledge about the factorial nature of the variables that he/she is able to specify that each measure xi depends only on a few of the factors j . If xi does not depend on j , ij = 0 in (16) (Slide 12). In many applications, the latent factor j represents a theoretical construct and the observed measures xi are designed to be indicators of this construct. In this case there is only one non-zero ij in each equation (16). Consider the situation where the same tests have been administered in G dierent groups and the factor analysis model applied in each group: xg = g g + g , g = 1, 2, . . . , G The covariance matrix in group g is g = g g g + 2 g Hypothesis of factorial invariance: 1 = 2 = = G & % & (30) % (29) (28)

Slide 18

Slide 20

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Factorial Invariance with Latent Means

S orbom (1974) extended the model in Slide 20 to include intercepts in (28): xg = g + g g + g , g = 1, 2, . . . , G Under complete factorial invariance: (31)

Econometric Models

yt = + Byt + xt + zt

(35)

Slide 21

Slide 23

t = 1, 2, . . . , N

(36)

yti = i + (i) yt(i) + (i) xt(i) + zti , 1 = 2 = = G 1 = 2 = = G (32) (33) = Cov % $ g and Sg be the sample mean vector and covariance matrix Let z in group g , and let g () and g () be the corresponding population mean vector and covariance matrix g = 1, 2, . . . , G. The t function for the multigroup case is dened as Ng Fg () , N g=1
G

(37)

he showed that one can estimate the mean vector and covariance matrix of in each group on a scale common to all groups. & ' & '

y x

A A + AA A

(38)

% $

Some History of LISREL

The idea of combining features of both econometrics and psychometrics into a single mathematical model was born in my mind in the spring of 1970. This idea was inspired by work of Professor Arthur S. Goldberger published in Psychometrika, 1971. The rst version of LISREL was a linear structural equation model for latent variables, each with a single observed, possibly fallible , indicator . I presented this model at the conference on Structural Equation Models in the Social Sciences held in Madison, Wisconsin, in November 1970. The proceedings of this conference, edited by Professors Goldberger and Duncan, were published in 1973. & %

Slide 22

F() =

(34)

Slide 24

zg , Sg , g ( ), g ()) is any of the t functions where Fg ( ) = F( dened for a single group. Here Ng is the sample size in group g and N = N1 + N2 + . . . + NG is the total sample size. To test the model, one can again use c = (N 1) times the minimum of F as a 2 with degrees of freedom d = Gk (k + 1)/2 t, where k is the number of variables. & %

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Slide 25

This LISREL model was generalized in 1971-72 to include models previously developed for multiple indicators of latent variables , for conrmatory factor analysis , for simultaneous factor analysis in several populations and more general models for covariance structures . The basic form of the LISREL model has remained the same ever since and is still the same model as used today. The general form of the LISREL model, due to its exible specication in terms of xed and free parameters and simple equality constraints, has proven to be so rich that it can handle not only the large variety of problems studied by hundreds of behavioral science researchers but also complex models, such as multiplicative MTMM models, non-linear models, and time series models, far beyond the type of models for which it was originally conceived. & ' The rst version of LISREL made generally available and with a written manual was LISREL III. It had xed column input, xed dimensions, only the maximum likelihood method, and users had to provide starting values for all parameters. The versions that followed demonstrated an enormous development in both statistical methodology and programming technology: % $

x1

k Q + k Q + +

x2

x3

Slide 27

x4

x5

x6

Q 1 Q Q11 J 1 Q 6 1 QQ s J 21 y) P P( (x) 3 1 J 12 21 P P Q32 q I J 6 Q ? J 1I 1 Q 2 21 (x) J 21 12 21 52 2 6 J ? 1 1 ^ J (y ) 31 32 2 P P42 3 23 P P q ? R 1 3 (x) 73

21 (x) 31

Q1 Q (x)

y1

y2

y3

y4

x7

The LISREL Model in LISREL Notation & ' % $

The LISREL Model with Means

y = y + y + x = x + x + Slide 28 = + B + + y, x = Observed Variables , = Latent Variables , = Measurement Errors = Structural Errors y , x , = Intercept Terms y , x , B, = Parameter Matrices

Slide 26

LISREL IV (1978) had Keywords, Free Form Input, and Dynamic Storage Allocation LISREL V (1981) had Automatic Starting Values, Unweighted and Generalized Least Squares, and Total Eects LISREL VI (1984) had Parameter Plots, Modication Indices, and Automatic Model Modication LISREL 7 (1988) had PRELIS, Weighted Least Squares, and Completely Standardized Solution LISREL 8 (1994) had SIMPLIS, Path Diagrams, and Non-linear Constraints

&

&

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

y + y (I B)1 ( + ) x + x

Assumptions
is uncorrelated with Slide 31
= y A( + )A y + x A y + where A = (I B)1 . The elements of and are functions of the elements of , , y , x , y , x , B, , , , , , and which are of three kinds: xed parameters that have been assigned specied values, constrained parameters that are unknown but linear or non-linear functions of one or more other parameters, and free parameters that are unknown and not constrained. y Ax + x x + ,

Slide 29

is uncorrelated with is uncorrelated with is uncorrelated with I B is non-singular and

& '

% $

& '

% $

LISREL as a Mean and Covariance Structure

Let Slide 30 = E () = Cov () = Cov = Slide 32 or

General Mean and Covariance Structures

Let and be functions of a parameter vector : = ( ) = ( ) (39)

= ( ) = () where is a vector of the non-duplicated elements of .

(40)

Then it follows that the mean vector and covariance matrix of z = (y , x ) are

&

&

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Some Formulas
Let s be a vector of the non-duplicated elements of S and assume that n 2 (s ) N (0, ) Denitions k = number of observed variables 1 k(k + 1) s = 2 t = number of independent parameters < s
1

Slide 35
st

= =

()
s1 t1

(41)

Slide 33

evaluated at

c sd

= orthogonal complement to c = 0 [|c ] non-singular

d = st & ' K
k s
2

% $ = D(D D)1 , where D is the duplication matrix

k s
2

& ' Fit Functions F = (s ) V (s ) ,

ss

% $

s
s1

= K vec(S) = nACov(s)

vec(S) = Ds n=N 1 Slide 36

where the weight matrix V is dened dierently for dierent t functions: ULS: V = I = diag (1, 2, 1, 2, 2, 1, . . .) GLS: V = D (S1 S1 )D ML: V = D (
1

ss

Slide 34

W
ss

= n Est[ACov(s)] )K under NT, = 2K ( = (wgh,ij ) under NNT, = nEst[ACov(sgh , sij )] = mghij sgh sij ,
N

WNT WNNT wgh,ij mghij &

)D if WNNT singular

1 WLS: V = WNNT

or WNNT

1 1 DWLS: V = D W = [diagW]

with

W = WNT W = WNNT

or

= (1/N )

a=1

(zag z g )(zah z h )(zai z i )(zaj z j ) % &

Karl G J oreskog '

Factor Analysis and Its Extensions $ Results '

Karl G J oreskog

Factor Analysis and Its Extensions $

E
ss

= V

What are the factors behind this development?

Models can be tested Slide 39 Computer technology Simple command language Path diagram SEM courses at many universities Journal of Structural Equation Modeling

) = E1 VWVE1 nACov( Slide 37 ) = W E1 nACov(s with c2 h1 c3 c4 W = WNT W = WNNT ) c (c WNT c )1 c (s ) = n(s = tr[(c WNT c )1 (c WNNT c )] = (d/h1 )c2 ) c (c WNNT c ) = n(s
1

or

) c (s % $ & '

& '

% $

The Growth of Structural Equation Modeling

350

The Success of Structural Equation Modeling

300

Slide 38

There has been an enormous development of structural equation modeling in the last 30 years. Proof: Thousands of journal articles Hundreds of dissertations Numerous books

250

Slide 40

200

150

100

50

1994 1995 1996 1997 1998 1999 2000 2001

Number of Journals and Articles by Year & % & %

Karl G J oreskog '

Factor Analysis and Its Extensions $ '

Karl G J oreskog

Factor Analysis and Its Extensions $

Main Virtues of SEM Methodology

SEM has the power to test complex hypotheses involving causal relationships among construct or latent variables SEM unies several multivariate methods into one analytic framework

Closing the Circle Back to Factor Analysis Latent Variable Models

Slide 43 f (x) = h()g (x | )d
p

Slide 41

SEM specically expresses the eects of latent variables on each other and the eect of latent variables on observed variables SEM can be used to test alternative hypotheses. SEM gives social and behavioral researchers powerful tools for stating theories more exactly, testing theories more precisely, generating a more thorough understanding of observed data.

(42)

f (x) =

h()
i=1

g (xi | )d

(43)

N (0, I) x | N ( + , 2 ) x = N (, + 2 ) (44) % $ & ' % $

& '

Binary and Ordinal Variables Retraction

xi = 1, 2, . . . mi Is it really that great? Slide 42 In the preface of his 1975 book O. D. Duncan said that he was fascinated by the formal properties of causal models but held a rather agnostic view of their utility. We have certainly come to great strides in the formal realm but are there really any great substantive applications? Slide 44
k (i) gi (xi = s | ) = F (s (i) (i) k

(45)
(i)

j =1

ij j ) F (s1

ij j )
j =1

(46)

(i) (i) < m = = 0 < 1 < 2 < m i 1 i t

1 2 1 e 2 u du 2

(i)

NOR : F (t) = (t) =

(47) (48) %

POM : F (t) = (t) = & % &

et 1 + et