Author kevinbinz

A Dual-Process Theory of Moral Judgment

kevinbinz Leave a comment

Part Of: Demystifying Ethics sequence
Content Summary: 900 words, 9 min read

Trolleyology

An ethical theory is an attempt to explain what goodness is: to ground ethics in some feature of the world. We have discussed five such theories, including:

  • Consequentialism, which claims that goodness stems from the consequences of an action
  • Deontology, which claims that goodness stems from absolute obligations (discoverable in light of the categorical imperative)

For most behaviors (e.g., theft), both theories agree (in this case, label the act as Evil). But there do exist key scenarios which prompt these theories to disagree. Consider the switch dilemma:

Consider a trolley barreling down a track that will kill five people unless diverted. However, on the other track a single person has been similarly demobilized. Should you pull to lever to divert the trolley?

Our ethical theories produce the following advice:

  • Consequentialism says: pull the lever! One death is awful, but better than five.
  • Deontology says: don’t pull the lever! Any action that takes innocent life is wrong. The five deaths are awful, but not your fault.

Would you pull the lever? Good people disagree. However, if you are like most people, you will probably say “yes”. 

Things get interesting if we modify the problem as follows. Consider the footbridge dilemma:

Consider a trolley barreling down a track that will kill five people unless diverted. You are standing on a bridge with a fat man. If you push the fat man onto the track, the trolley will derail, sparing the five people.

Notice that the consequences for the action remain the same. Thus, consequentialism says “push the fat man!”, and deontology says “don’t push!”

What about you? Would you push the fat man? Good people disagree; however, most people confess they would not push the fat man off the bridge. 

Our contrasting intuitions are quite puzzling. After all, they only thing that’s different between the two cases is how close the violence is: far away (switch) vs up close (shoving).

Towards a Dual-Process Theory

Recall that there are two kinds of ethical theories.

  1. Prescriptive theories describe what goodness objectively is.
  2. Descriptive theories tell us how the brain produces moral judgments.

Consequentialism and deontology are prescriptive theories. However, we can also conceive of consequentialism and deontology as descriptive theories. Let’s call these descriptive variants folk consequentialism and folk deontology, respectively.

Sometimes, people’s judgments is better explained by folk consequentialism; other times, folk deontology enjoys more predictive success. We might entertain two hypotheses to explain this divergence:

  1. Different boundary conditions of a single neural process
  2. Two competing processes for moral judgment

As we will see, the evidence suggests that the second hypothesis, the dual-process theory of moral judgment, is correct.

Dissociation-Based Evidence

Consider the crying baby dilemma:

It’s wartime. You and your fellow villagers are hiding from nearby enemy soldiers in a basement. Your baby starts to cry, and you cover your baby’s mouth to block the sound. If you remove your hand, your baby will cry loudly, and the soldiers will hear. They will find you… and they will kill all of your. If you do not remove your hand, your baby will smother to death. Is it morally acceptable to smother your baby to death in order to save yourself and the other villagers?

Here, people take a long time to answer, and show no consensus in their answers. If the dual-process theory of moral judgment is correct, then we expect the following:

  1. Everyone exhibits increased activity in the dorsal anterior cingulate (dACC). This region is known to reliably respond when two or more incompatible behavioral responses are simultaneously activated. 
  2. For those who eventually choose the folk consequentialist answer (save the most lives) should exhibit comparatively more activity in brain regions associated with working memory and cognitive control.

Both predictions turn out to be true. Here then is the circuit diagram of our dual-process, organized in the two cybernetic loops framework:

Dual-Process Morality- System Architecture

Four other streams of evidence corroborate our dual-process theory:

  • Deontological judgments are produced more quickly than consequentialist ones.
  • Cognitive distractions slow down consequentialist but not deontological judgments.
  • Patients with dementia or lesions that cause “emotional blunting” are disproportionately likely to approve of consequentialist action. 
  • People who are either high in “need for cognition” and low in “faith in intuition”, or have unusually high working memory capacity, tend to produce more consequentialist judgments.

Relation To Other Disciplines

We have previously distinguished two kinds of moral machinery:

  • Propriety frames are a memory format that retains social intuitions.
  • Social intuition generators which contribute to the contents of social judgments.

These machines map to the dual-process theory of judgment. Propriety frames are housed in cerebral cortex, which perform folk consequentialist analysis. Social intuition generators are located within the limbic system, and contribute folk deontology intuitions.

Recall that Kantian deontology attempted to ground moral facts in pure reason (the categorical imperative). While surely valuable as a philosophical exercise, in practice folk deontological judgments have little to do with reason. They are instead driven by autonomic emotional responses. It is folk consequential judgments which depend more on reason (cortical reasoning).

This is not to say that people who prefer consequential reasoning are strictly superior moral judges. But I will address the question which reasoning system should I trust more? on another day.

Takeaways

  • For the switch dilemma, most people reason consequentially (“save the most lives”)
  • For the footbridge dilemma, most people reason deontologically (“murder is always wrong”)
  • These contrasting styles emerge because the brain has two systems of judgment.
  • Folk consequentist reasoning is performed in cerebral cortex.
  • Folk deontology intuitions are generated from within the limbic system.

Until next time.

An Introduction to Ethical Theories

kevinbinz Leave a comment

Part Of: Demystifying Ethics sequence
Content Summary: 1500 words, 15 min

Motivations

Ethics discussions can occur at different levels of abstraction. The most concrete ethical discussions involve discussing specific topics; e.g., “is abortion moral”? These scenarios are called applied ethics.

Spend enough time wrestling with applied ethics, and you’ll start wondering what all of these moral intuitions have in common. An ethical theory is an answer to the question “what makes a thing good or bad”?

Meta-ethics answers a yet broader set of questions. What is right and wrong, anyways? Are moral beliefs objectively true in the same sense that the Earth orbits the sun? Or is my condemning an unethical act merely expressing an aesthetic preference (“yuck!”)?

Ethical Theory- Three Abstraction Levels

Image Credit

Today we conduct a whirlwind tour of ethical theories, touching on the most influential theories. Ethical theories are normative: they speak to what moral conclusions ought to be. However, we will see later how these theories also sometimes imperfectly anticipate the emerging science of moral cognition.

We will discuss the following:

  1. Divine Command Theory
  2. Natural Law Theory
  3. Virtue Theory
  4. Deontology
  5. Utilitarianism

The first three theories are the oldest, and have close ties to religious doctrine. The last two are the most well-known theories these days.

Divine Command Theory

Historically, religions have had a good deal to say about right and wrong. One of the oldest ethical theories is Divine Command Theory, which states:

Moral behaviors are those that are commanded (or willed) by the divine.

This theory was originally proposed in the context of Greek mythology. But since it doesn’t say much about the specific character of the divine, it was adopted by Augustine and some (but not all) modern theologians.

One of the strongest criticisms to Divine Command Theory is the Euthyphro Dilemma, which goes as follows:

  • Is an action ethical because God commands it?
  • Or does God command it because it is ethical?

Clearly, we must affirm one of these options. But which one? Either “horn” of the dilemma extracts painful concessions:

  • On the first option, moral content becomes arbitrary. If God changes his mind about moral facts, our moral universe is turned upside down.
  • On the second, God becomes a middleman who conveys independent ethical facts. Why can’t we simply learn moral truths direct from the source?

These days, most philosophers find Divine Command Theory problematic. But there are other ethical theories available to the faithful.

Natural Law

Another approach to ethical theories, championed by Thomas Aquinas, is Natural Law Theory. Moral truths are grounded as follows:

Moral behaviors are those which promote, and do not detract from, a basic good.

A basic good is injected into human nature by God. Here are three such lists, generated by a Catholic, a Muslim, and a philosopher:

Ethical Theories- Natural Law Basic Goods (7)

Ethical actions promote the basic goods; immoral actions detract from the basic goods. This, incidentally, is why Catholics oppose all forms of birth control: they detract from the basic good of Reproduction.

How did these philosophers generate their lists of basic goods? By observing nature. For example, Life and Reproduction usually make the cut because humans naturally prioritize survival and reproductive success (this is a consequence of natural selection).

However, there is a conceptual difficulty in deriving basic goods from nature. This was clearly expressed by Hume’s Law, also called the is-ought gap: what logical procedure could possibly be used to infer normative statements (an “ought”) from descriptive facts of the words (an “is”)?

Virtue Theories

Unlike its competitors, virtue theory doesn’t spend much time telling you what to do (moral behaviors). Instead, people are the subject of moral truths. Behaviors are only good or evil insofar as they enhance or corrupt a person’s character.

Aristotle presented these ideas in Nichomachean Ethics. There he describes ethical behavior as a way to achieve eudaimonia (human flourishing or “the good life”).

A life of eudaimonia is a life of striving. It’s a life of pushing yourself to your limits, and finding success. A eudaimonistic life will be full of the happiness that comes from achieving something really difficult, rather than just having it handed to you.

Aristotle also believed that moral knowledge is learned not by navel-gazing, but instead comes from social interactions. Specifically, good character is developed by spending time with virtuous people. This theory is where the idea of a role model comes from.

So let’s get down to business. In virtue theory, moral facts are grounded as follows:

Moral people are those who have character; that is, they possess basic virtues.

Aristotle identified twelve basic virtues: Courage, Moderation, Flexibility, Cleanliness, Generosity, Ambition, Patience, Truthfulness, Wittiness, Friendliness, Modesty, and Righteous Indignation. A good person nurtures these virtues.

The Golden Mean provides some insight into how these virtues can be refined over time. A person can respond too strongly with a virtue (excess), or too weakly (deficiency). For example, one must not have too much courage, nor too little:

Ethical Theory- Golden Mean (1)

Virtue ethics is surprisingly robust to philosophical criticism. However, one scientific concern has been gaining traction recently. Empirical research in the field of situational psychology increasingly suggests that there is no such thing as character traits.

Make no mistake, seeing vices and virtues in others is a universal human instinct. But there is no guarantee that mental software works correctly (e.g., see how the brain lies to itself about visual experience). And the fact of the matter is, character information contributes zero predictive power over how a person will respond to a given situation.

Duty Theories

The first ethical theories that isn’t explicitly theistic, Immanuel Kant’s deontological (“knowledge of what is proper”) system purports to result from pure reason.

Moral behaviors are those which obey the categorical imperative.

For Kant, the categorical imperative is a procedure by which a person can discover the rigid contents of the moral universe. He presents four perspectives that independently arrive at these moral truths; the two most influential of which are:

  1. The universalizability principle instructs us to act only if you also desire that your action should become a universal law. Lying is wrong because if lying becomes universalized, meaningful communication becomes impossible.
  2. The principle of humanity instructs us to act in such a way that you treat humanity (both yourself or someone else), always also as an end and never simply as a means. This principle underlies the modern idea of human rights, and is leveraged by Nozick’s brand of libertarianism. 

The categorical imperative promotes a kind of moral absolutism: if actions are right or wrong intrinsically, their moral status is invariant to the consequences of an action. And this inflexibility can produce counterintuitive judgments.

For example, consider a house shielding a Jewish family from the Nazis. If a soldier comes to the door asking about its residents, most of use think it would be morally acceptable to lie. Kant, however, claims that lying is wrong in all situations, even ones such as these.

Consequentialist Theories

Deontological theories are invariant to behavioral outcome. But there are many moral situations where outcome seems to matter. Consider the trolley problem, where a trolley is currently barreling down a track and should kill five people unless diverted. However, on the other track one person has been similarly demobilized:

Ethical Theories- Trolley Problem (1)

Kant would say that pulling the lever is simply murder. That five people die is regrettable, but their death does not incriminate the person at the switch. If the person does divert the trolley, the guilt of the single person’s death is on his head.

However, most people think that the number of deaths – the consequence of an action – matters. Consequentialist theories make this intuition explicit:

Moral behaviors are those that bring the most good to the most people.

Consequentialist theories vary on how they define “the most good” (personal well-being) and “the most people” (societal well-being). 

For example, classical utilitarianism, advocated by Jeremy Bentham and John Stuart Mill, which takes the following stances:

  1. Personal well-being originates from hedonism: actions are moral if they increase pleasure and/or decrease pain.
  2. Societal well-being is taken as the sum total of the well-being of constituent individuals.

Many people condemn utilitarianism for its hedonism (“don’t immoral behaviors often feel pleasurable?”). However, this criticism loses some of its force when you remember to distinguish higher pleasures (eg., aesthetic experience) versus baser pleasures. 

The problem of figuring out societal well-being, how to optimize outcomes for communities, is much more challenging. The repugnant conclusion, aka the mere addition paradox, is a particularly damning criticism. A large society with barely tolerable quality of life (e.g., urban sweat shops) doesn’t feel morally equivalent to happier, smaller societies. 

Ethical Theories- Repugnant Conclusion

Thus, maximizing total well-being may be problematic. But different problems emerge if we instead optimize against average well-being. To this day, designing aggregation functions to compute societal well-being remains an essentially unsolved problem.

Takeaways

Today, we discussed the following theories:

Ethical Theories- Summary

There exist other theories besides these, of course. For example, Hobbes’ contractualism makes important contributions to the conversation. 

To date, no ethical theory has emerged as a clear favorite. A recent survey of professional philosophers reveals considerable support for three of the above theories:

Ethical Theories- Philosopher Opinions

Next time, we will explore whether we should view these normative theories as competitors, or complementary solutions to different concerns. Until then.

An Overview of Moral Cognition

kevinbinz 2 Comments

Part Of: Demystifying Ethics sequence
Content Summary: 1000 words, 10 min read

Propriety Frames

Main Article: An Introduction To Propriety Frames

The primate brain contains a diverse set of memory structures. For example, episodic and semantic memory store narratives and facts, respectively.

Propriety frames are a memory format that retains social intuitions. This form of memory permits normative judgments, or which behaviors are appropriate or inappropriate.

Propriety frames are organized by situational context. A Restaurant frame provides the social intuitions for restaurant interactions. Frames are the substrate of rituals.

Frames are organized hierarchically to promote reuse. For example, the Eating frame is relevant in contexts besides restaurant dining.

Propriety Frames- Restaurant Example (2)

When a mother instructs her son to not yell in the store, the child installs an update to his Shopping frame. When a family exchanges gossip around a campfire, they are doing so in part to synchronize their propriety frames.

Intuition Generators

Main Article: Moral Foundations Theory

The contents of our social intuitions is not arbitrary. Our environment does not fully determine our moral sense. The brain also possesses innate social intuition generators, which contribute to the contents of social judgments.

As outlined in Moral Foundations theory, there are six such generators: authority, care, loyalty, fairness, autonomy, and sanctity.

People are genetically and environmentally disposed to respond to certain generators more strongly than others. Let social matrices be the emotional intensities elicited by each generator.

People with similar matrices tend to gravitate towards similar political parties. When you measure the social matrices of American citizens, you can see large differences between the social intuitions of Democrats and Republicans.

Theory of Normatives- Social Matrices by Political Party (1)

These differences in social matrices explain much of American politics.

  • Why do Democrats praise entitlements, but Republicans denounce them? Because Democrats heavily emphasize Care for the poor, whereas Republicans more strongly reverberate to questions of Fairness (exploitation and moral hazard).
  • Why are Democrats more skeptical of patriotism than their Republican counterparts? Perhaps because they respond to Loyalty to country less.

For more information, see Graham et al (2009). Liberals and conservatives rely on different sets of moral foundations.

Generators vs Frames

Main Article: A Dual-Process Theory of Moral Judgment

Two mechanisms contribute to social intuitions: propriety frames, and intuition generators.

Theory of Normatives- Frame vs Generator (1)

Generators are located subcortically, and thus produce intuitions quickly and automatically. Frames are stored in the cortical mantle, and are thus slower, and more amenable to conscious awareness.

As the work of Joshua Greene shows, deontological (rule-based) attitudes are generated rapidly; whereas consequential theories of morality are slower and more vulnerable to distraction. This suggests a relatively straightforward mapping:

  • People with a deontological style rely more heavily on their Generators.
  • People with a consequentialist style lean instead on their Frames.

Frames and Generators can influence one another, albeit slowly. Moral argumentation can change one’s mind, and these frame updates can percolate down to change one’s social matrices (intuition weights). Call this frame internalization.

Similarly, if one’s private intuitions diverge from a culturally inherited norm, that frame can be updated to be more consistent with one’s personality. Call this frame refinement.

For more information, see Cushman, Young, Greene (2010). Our multi-system moral psychology: Towards a consensus view

Cultural Regimes

Main Article: The Relational Sphere Hypothesis

Human communities generate emergent networks known as cultural regimes. These emerge as distinct categories of frames and intuition generators.

regime categories

Language made regime accretion possible. No longer is expertise confined to the skull of the individual. No longer is death a Full Restart button. Our species sends cultural expertise down through the generations.  This knowledge – these frames – have become increasingly sophisticated over the course of human history.

Hominid evolution has also seen the advent of regime diversification. Modern religions (of the Axial Age) were successful because they improved on our ability to trust strangers. This in turn dramatically increased the size of feasible social groups.

Theory of Normativity- Evolution of Regimes (3)

Moral Tagging

We have so far only been speaking about social attitudes (should / should not). What about moral attitudes (good / evil)?

Moral attitudes are nothing more than a kind of social attitude. I know of no moral attitude that can be divorced from a social context.

Moral attitudes are constructed by moral tagging, which endows a subset of social attitudes with anger and disgust reactions (as opposed to the more typical reputation appraisal, gossiping, shaming).

Further, moral tagging produces appraisal inflexibility. Moral violations are viewed as wrong everywhere, in every context. This is in contrast to other social violations, for which it is easier to see circumstances in which the behavior would seem less bad.

The boundary between Virtuous and Tolerable is smooth, reflecting the flexibility of our intuition generators. In contrast, the boundary between Tolerable and Intolerable is sharp.

Moral Flinch- Latitudes of Acceptance (2)
A similar distinction appears amidst disagreement. There are two kinds of ways people’s judgments can diverge:

  • Moral disagreement: is a given behavior Evil?
  • Social disagreement: is a given behavior Inappropriate?

moral vs social disagreement

For more information, see Tetlock et al (2000). The psychology of the unthinkable: taboo trade-offs, forbidden base rates, and heretical counterfactuals.

Norm Synchronization

The space of possible social intuitions is vast. However, group living only became possible with relatively homogenous norms. How do individual brains synchronize propriety frames and social matrices within a group?

At least four mechanisms provide norm synchronization.

  • Natural language and facial expressions are used to communicate social and emotional information.
  • Reputation systems benefit social beings who are especially adept at learning & conforming to the frames of their peers.
  • First-order punishment emotions (anger at transgressor) incentivizes people to not violate implicit societal expectations.
  • Second-order punishment (anger at those who are accepting of transgressor) incentivize a community to respond to, and develop a unified response to, dyadic disagreements.

Takeaways

  • In Propriety Frames, we saw how the brain retains social information.
  • Limbic machinery, such as the Care module, generate normative intuitions.
  • Generators and frames interact to facilitate both top-down and bottom-up learning.
  • A subset of social violations are tagged as morally intolerable.
  • Several emotion devices work to consolidate normatives within a group. 

Related Work

  • Graham, Haidt, Nosek (2009). Liberals and conservatives rely on different sets of moral foundations.
  • Cushman, Young, Greene (2010). Our multi-system moral psychology: Towards a consensus view
  • Tetlock et al (2000). The psychology of the unthinkable: taboo trade-offs, forbidden base rates, and heretical counterfactuals.

[Sequence] Psychology Musings

kevinbinz Leave a comment

This sequence is simply a bucket that collects some of my older, “pre-theoretic” views on psychology. YMMV. 🙂

Random musings

Books I enjoyed

[Sequence] Cognitive Architecture

kevinbinz Leave a comment

Central Organizing Principles

Main Sequence

Version Iterations

  • Version 0.1: The mind as a collection of functional modules.
  • Version 1.0: Integrating anatomy and genetics.
  • Version 1.1: Integrating ecology, morality, frame memory, and social systems.
  • Version 1.2: Integrating theory of mind, social reasoning and culture.
  • Version 1.3: Integrating memory hierarchy, motivation, frame memory.
  • Version 1.4: Integrating working memory, attention, consciousness.
  • Version 1.5: Integrating cognitive control, affective neuroscience.

Singular Value Decomposition

kevinbinz Leave a comment

Part Of: Algebra sequence
Followup To: Eigenvalues and Eigenvectors
Content Summary: 1300 words, 13 min read.

Limitations of Eigendecomposition

Last time, we learned how to locate eigenvalues and eigenvectors of a given matrix. Diagonalization is the process where a square matrix can be decomposed into two factors: a matrix Q (with eigenvectors along the columns) and a matrix \Lambda (with eigenvalues along the diagonal).

A = Q \Lambda Q^T

But as we saw with the spectral theorem, eigendecomposition only works well against square, symmetric matrices. If a matrix isn’t symmetric, it is easy to run into complex eigenvalues. And if a matrix isn’t square, you are out of luck entirely!

Eigendecomposition- Spectral Theorem (1)

Can we generalize eigendecomposition to apply to a wider family of matrices? Can we diagonalize matrices of any size, even if they don’t have “nice” properties?

Yes. Self-transposition is the key insight of our “eigendecomposition 2.0”. We define the self-transpositions of A as AA^{T} and A^{T}A.

Suppose A \in \mathbb{R}^{m x n}. Then AA^T \in \mathbb{R}^{mxm} and A^TA \in \mathbb{R}^{nxn}. So these matrices are square. But they are also symmetric!

To illustrate, consider the following.

A = \begin{bmatrix}  4 & 4 \\ -3 & 3 \\ \end{bmatrix}

Since A is not symmetric, we have no guarantee that its eigenvalues are real. Indeed, its eigenvalues turn out to be complex:

\det(A - \lambda I) = \begin{bmatrix} 4 - \lambda & 4 \\ -3 & 3 - \lambda \\ \end{bmatrix} = 0

(12 - 7 \lambda + \lambda^2) + 12 = 0 \Rightarrow \lambda^2 -7 \lambda + 24 = 0

\lambda = \frac{7 \pm \sqrt{(-7)^2 - 4*1*24}}{2*1} = \frac{7}{2} \pm \frac{\sqrt{47}i}{2}

Eigendecomposition on A sucks. Are the self-transposed matrices any better?

A^TA = \begin{bmatrix} 4 & -3 \\ 4 & 3 \\ \end{bmatrix} \begin{bmatrix} 4 & 4 \\ -3 & 3 \\ \end{bmatrix} = \begin{bmatrix} 25 & 7 \\ 7 & 25 \\ \end{bmatrix}

AA^T = \begin{bmatrix} 4 & 4 \\ -3 & 3 \\ \end{bmatrix} \begin{bmatrix} 4 & -3 \\ 4 & 3 \\ \end{bmatrix} = \begin{bmatrix} 32 & 0 \\ 0 & 18 \\ \end{bmatrix}

These matrices are symmetric! Thus, they are better candidates for eigendecomposition.

Towards Singular Value Decomposition

Singular Value Decomposition (SVD) is based on the principle that all matrices are eigendecomposable after self-transposition. It is essentially a bug fix:

SVD- Self-Transposition

An important way to picture SVD is with the idea of orthogonal bases. It is relatively easy to find any number of orthogonal bases for a given rowspace. Call the matrix of orthogonal vectors V.

We desire to find an orthogonal basis V such that AV produces an orthogonal basis in the column space. Orthogonal bases are not particularly hard to find. But most orthogonal bases, once projected to column space, will lose their orthogonality! We desire that particular orthogonal basis U such that V is also orthogonal.

We won’t require vectors in U to be the same size as those in V. Instead, we will normalize V; its basis vectors will be orthonormal (orthogonal and normal). Then the length of each vector in U will diverge by a scaling factor.

As we will soon see, these scaling factors are not eigenvalues. Instead, we will use sigmas instead of lambdas.

  • Scaling factors \sigma , analogous to eigenvalues \lambda.
  • Diagonal matrix \Sigma , analogous to diagonal matrix \Lambda

Our full picture then, looks like this:

SVD- Orthogonal Basis Transfer

Let us now translate this image into matrix language.

A \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ v_1 & v_2 & \dots & v_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix} = \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ \sigma_1u_1 & \sigma_2u_2 & \dots & \sigma_nu_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix}

But we can easily factorize the right-hand side:

\begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ \sigma_1u_1 & \sigma_2u_2 & \dots & \sigma_nu_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix} = \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ u_1 & u_2 & \dots & u_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix} * \begin{bmatrix} \sigma_1 & 0 & \dots & 0 \\ 0 & \sigma_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \sigma_n \\ \end{bmatrix}

So we have that:

AV = U \Sigma

Since both V and U are orthogonal, inversion of either is equivalent to transposition:

A = U \Sigma V^{-1} = U \Sigma V^T

This strongly resembles our diagonalization equation A = Q \Lambda Q^T. SVD distinguishes itself by considering two orthogonal eigenmatrices U and V, not just one (Q).

Recalling that A = U \Sigma V^T ,

A^TA = (V \Sigma^T U^T) (U \Sigma V^T)

But now the innermost term cancels. Since \Sigma is a square diagonal matrix, its self-transposition is simply equal to \Sigma^{2}. So,

A^TA = V \Sigma^2 V^T

Since A^{T}A is a square, symmetric matrix, our diagonalization theorem applies!

A^TA = V \Sigma ^2 V^T = Q \Lambda Q^T

To find U, a similar trick works:

AA^T =  (U \Sigma V^T)(V \Sigma^T U^T) = U \Sigma^2 U^T = Q \Lambda Q^T

The relationships between SVD and eigendecomposition are as follows:

  • V is the eigenvectors of A^TA
  • U is the eigenvectors of AA^T
  • \Sigma is the square root of the eigenvalues matrix \Lambda

If any eigenvalue is negative, the corresponding sigma factor would be complex. But A^TA and AA^T are positive-semidefinite, which guarantees non-negative eigenvalues. This assures us that \Sigma contains only real values.

In contrast to eigendecomposition, every matrix has an SVD decomposition. Geometrically, V and U act as rotational transformations, and Sigma acts as a scaling transformation. In other words, every linear transformation comprises a rotation, then scaling, then another rotation.

SVD- Rotation Scaling Rotation (1)

A Worked Example

Let’s revisit A. Recall that:

A = \begin{bmatrix} 4 & 4 \\ -3 & 3 \\ \end{bmatrix}, A^TA = \begin{bmatrix} 25 & 7 \\ 7 & 25 \\ \end{bmatrix}, AA^T = \begin{bmatrix} 32 & 0 \\ 0 & 18 \\ \end{bmatrix}

Eigendecomposition against A is unpleasant because A is not symmetric. But A^{T}A is guaranteed to be positive semi-definite; that is, to have non-negative eigenvalues. Let’s see this in action.

det(A^TA - \lambda I) = \begin{bmatrix} 25 - \lambda & 7 \\ 7 & 25 - \lambda \\ \end{bmatrix} = 0

\lambda^2 - 50 \lambda + 576 = 0 \Rightarrow (\lambda_1 - 32)(\lambda_2 - 18) = 0

\lambda_1 = 32, \lambda_2 = 18

trace(A^TA) = 50 = \sum{\lambda_i}

det(A^TA) = 625-49 = 576 = 18 * 32 = \prod{\lambda_i}

These are positive, real eigenvalues. Perfect! Let’s now derive the corresponding (normalized) eigenvectors.

A - 32I = \begin{bmatrix} -7 & 7 \\ 7 & -7 \\ \end{bmatrix} = 0, A - 18I = \begin{bmatrix} 7 & 7 \\ 7 & 7 \\ \end{bmatrix} = 0

rref(A - 32I) = \begin{bmatrix} -1 & 1 \\ 0 & 0 \\ \end{bmatrix} = 0, rref(A-18I) = \begin{bmatrix} 1 & 1 \\ 0 & 0 \\ \end{bmatrix} = 0

v_1 = \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \\ \end{bmatrix}, v_2 = \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \\ \end{bmatrix}

SVD intends to decompose A into U \Sigma V^{T}. The above findings give us two of these ingredients.

V^{T} = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \end{bmatrix}

\Sigma =\begin{bmatrix} \sqrt{32} & 0 \\ 0 & \sqrt{18} \\ \end{bmatrix}

What’s missing? U! To find it, we perform eigendecomposition on AA^{T}. This is an especially easy task, because AA^{T} is already a diagonal matrix.

AA^T = \begin{bmatrix} 32 & 0 \\ 0 & 18 \\ \end{bmatrix}

U = \begin{bmatrix} u_1 & u_2 \\ \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}

We have arrived at our first Singular Value Decomposition.

A = U \Sigma V^T = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix} \begin{bmatrix} \sqrt{32} & 0 \\ 0 & \sqrt{18} \\ \end{bmatrix} \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & - \frac{1}{\sqrt{2}} \\ \end{bmatrix}

Okay, so let’s check our work. 😛

A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix} \left( \begin{bmatrix} \sqrt{32} & 0 \\ 0 & \sqrt{18} \\ \end{bmatrix} \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & - \frac{1}{\sqrt{2}} \\ \end{bmatrix} \right) = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix} \begin{bmatrix} 4 & 4 \\ 3 & 3 \\ \end{bmatrix} = \begin{bmatrix} 4 & 4 \\ 3 & 3 \\ \end{bmatrix}

These matrices are a viable factorization: multiplication successfully recovers A.

Takeaways

  • Eigendecomposition only works for a subclass of matrices; SVD decomposes all matrices.
  • SVD relies on self-transposition to convert any arbitrary matrix into one that works well against eigendecomposition (guarantees square m = n and symmetric A = A^{T}).
  • Another way to interpret SVD is by taking a special kind of orthogonal basis that, once passed through the linear transformation, preserves its orthogonality.
  • Every matrix A = U \Sigma V^{T}. That is, every linear transformation can be conceived as rotation + scaling + rotation.

Until next time.

Eigenvalues and Eigenvectors

kevinbinz 1 Comment

Part Of: Algebra sequence
Followup To: An Introduction to Linear Algebra
Next Up: Singular Value Decomposition
Content Summary: 1300 words, 13 min read

Geometries of Eigenvectors

Matrices are functions that act on vectors, by mapping from row-vectors to column-vectors.  Consider two examples:

  1. Reflection matrices, which reflect vectors across some basis.
  2. Rotation matrices, which rotate vectors clockwise by \theta degrees.

eigenvectors-transformation-geometries-1

The set of eigenvectors of a matrix A is a special set of input vectors for which the matrix behaves as a scaling transformation. In other words, we desire the set of vectors \vec{x} whose output vectors A\vec{x} differ by a scaling factor.

Eigenvectors have a straightforward geometric interpretation:

  1. Reflection eigenvectors are orthogonal or parallel to the reflecting surface. In the left image above, that is the top two pairs of vectors.
  2. Rotation eigenvectors do not exist (more formally, cannot be visualized in \mathbb{R}^2).

Algebra of Eigenvectors

We can express our “parallel output” property as:

A\vec{x} = \lambda \vec{x}

Thus \vec{x} and A\vec{x} point in the same direction, but differ by scaling factor \lambda.

Scaling factor \lambda is the eigenvalue. There can be many \left( x, \lambda \right) pairs that satisfy the above equality.

For an \mathbb{R}^{n x n} matrix, there are n eigenvalues. These eigenvalues can be difficult to find. However, two facts aid our search:

  • The sum of eigenvalues equals the trace (sum of values along the diagonal).
  • The product of eigenvalues equals the determinant.

To solve, subtract \lambda \vec{x} from both sides:

A\vec{x} = \lambda \vec{x}

(A - \lambda I)\vec{x} = 0

We would like to identify n unique eigenvectors. But if the new matrix (A - \lambda I) has an empty nullspace, it will contain zero eigenvectors. So we desire this new matrix to be singular.

How to accomplish this?  By finding eigenvalues that satisfy the characteristic equation \det(A - \lambda I) = 0. Matrices are singular iff their determinants equal zero.

Let’s work through an example! What is the eigendecompositon for matrix A:

A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \\ \end{bmatrix}

We need to find eigenvalues that solve the characteristic equation.

\det(A - \lambda I) = \begin{vmatrix} 3-\lambda & 1 \\ 1 & 3-\lambda \\ \end{vmatrix} = 0

(3 - \lambda)^2 - 1^2 = \lambda_2 -6\lambda + 8 = (\lambda-2)(\lambda-4) = 0

\lambda_1 = 2, \lambda_2 = 4

Are these eigenvalues correct? Let’s check our work:

trace(A) = 6 = \sum{\lambda_i}

det(A) = 8 = \prod{\lambda_i}

How to find our eigenvectors? By solving the nullspace given each eigenvalue.

For \lambda_1=2 :

A - 2I = \begin{bmatrix} 1 & 1 \\ 1 & 1 \\ \end{bmatrix} \Rightarrow rref(A - 2I) = \begin{bmatrix} 1 & 1 \\ 0 & 0 \\ \end{bmatrix}

(\lambda_1, \vec{x}_1) = (2, \begin{bmatrix} 1 \\ -1 \\ \end{bmatrix})

For \lambda_2=4 :

A - 4I = \begin{bmatrix} -1 & 1 \\ 1 & -1 \\ \end{bmatrix} \Rightarrow rref(A - 4I) = \begin{bmatrix} -1 & 1 \\ 0 & 0 \\ \end{bmatrix}

(\lambda_2, \vec{x}_2) = (4, \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix})

Desirable Matrix Properties

The above example was fairly straightforward. But eigendecomposition can “go awry”, as we shall see. Consider a rotation matrix, which in two dimensions has the following form:

R = \begin{bmatrix} \cos(\theta) & -\sin(\theta) \\ \sin(\theta) & \cos(\theta) \\ \end{bmatrix}

What are the eigenvalues for rotation \theta = 90^{\circ} ?

R = \begin{bmatrix} 0 & -1 \\ 1 & 0 \\ \end{bmatrix}

\det(R - \lambda I) = \begin{bmatrix} - \lambda & -1 \\ 1 & - \lambda \\ \end{bmatrix} = 0

(- \lambda)^2 - 1^2 \Rightarrow \lambda^2 = 1

\lambda_1 = i, \lambda_2 = -i

We can check our work:

trace(R) = 0 = \sum{\lambda_i}

det(R) = 1 = \prod{\lambda_i}

We saw earlier that rotation matrices have no geometric interpretation. Here, we have algebraically shown that its eigenvalues are complex.

A = \left[ \begin{smallmatrix} 3 & 1 \\ 1 & 3 \\ \end{smallmatrix} \right] has real eigenvalues, but R = \left[ \begin{smallmatrix} 0 & -1 \\ 1 & 0 \\ \end{smallmatrix} \right] has less-desirable complex eigenvalues.

We can generalize the distinction between A and R as follows:

Spectral Theorem. Any matrix that is symmetric (A = AT) is guaranteed to have real, nonnegative eigenvalues. The corresponding n eigenvectors are guaranteed to be orthogonal.

In other words, eigendecomposition works best against symmetric matrices.

Eigendecomposition- Spectral Theorem (1)

Diagonalization

Let us place each eigenvector in the column of a matrix S. What happens when you multiply the original matrix A by this new matrix? Since S contains eigenvectors, multiplication by A reduces to multiplication by the associated eigenvalues:

AS = \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ \lambda_1x_1 & \lambda_2x_2 & \dots & \lambda_nx_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix}

We see the product contains a mixture of eigenvalues and eigenvectors. We can separate these by “pulling out” the eigenvalues into a diagonal matrix. Call this matrix \Lambda (“capital lambda”).

AS = \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ x_1 & x_2 & \dots & x_n \\ \vdots & \vdots & \vdots & \vdots \\ \end{bmatrix} * \begin{bmatrix} \lambda_1 & 0 & \dots & 0 \\ 0 & \lambda_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \lambda_n \\ \end{bmatrix} = S \Lambda

Most matrices have the property that its eigenvectors are linearly independent. For such matrices, S is invertible. Given this fact, we can solve for A:

\Lambda = S^{-1} A S

A = S \Lambda S^{-1}

Matrices that can be factorized in this way are said to be diagonalizable. We can see that both elimination and eigendecomposition are performing the same type of work: factorizing matrices into their component parts.

If A is symmetric, then we know \Lambda is real, and its eigenvectors in S are orthogonal. Let us rename S to be Q, to reflect this additional property. But orthogonal matrices have the property that transposition equats inversion: Q^T = Q^{-1}. Thus, if A is symmetric, we can simplify the diagonalization formula to:

A = Q \Lambda Q^{-1} = Q \Lambda Q^T

Asymptoptic Interpretations

This diagonalization approach illustrates an important use case of eigenvectors: power matrices. What happens when A is applied arbitrarily many times? What does the output look like in the limit?

We can use the diagonalization equation to represent A^k:

A^k = \prod_{i=1}^{k} (Q^{-1} \Lambda Q) = (Q^{-1} \Lambda Q)(Q^{-1} A Q)\dots(Q^{-1} \Lambda Q)

We can simplify by canceling the inner terms QQ^{-1}:

A^k = Q^{-1} \Lambda^k Q

This equation tells us that the eigenvectors is invariant to how many times A is applied. In contrast, eigenvalue matrix \Lambda has important implications for ongoing processes:

  • If each eigenvalue has magnitude less than one, the output will trend towards zero.
  • If each eigenvalue has magnitude greater than one, the output will trend to infinity.

Fibonacci Eigenvalues

The powers interpretation of eigenvalues sheds light on the behavior of all linear processes. This includes number sequences such as the Fibonacci numbers, where each number is the sum of the previous two numbers.

Recall the Fibonacci numbers are 0,1,1,2,3,5,8,13,... What is F_{100} ?

Eigenvalues can answer this question. We must first express the Fibonacci generator as a linear equation:

F(k+2) = 1F(k+1) + 1F(k)

In order to translate this into a meaningful matrix, we must add a “redundant” equation

F(k+1) = 1F(k+1) + 0F(k)

With these equations, we can create a 2×2 Fibonacci matrix F.

F = \begin{bmatrix} 1 & 1 \\ 1 & 0 \\ \end{bmatrix}

This matrix uniquely generates Fibonacci numbers.

u_1 = Fu_0 = \begin{bmatrix} 1 & 1 \\ 1 & 0 \\ \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}

u_4 = F^4u_0 = \begin{bmatrix} 1 & 1 \\ 1 & 0 \\ \end{bmatrix}^4 \begin{bmatrix} 1 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 5 \\ 3 \\ \end{bmatrix}

To discover the rate at Fibonacci numbers grow, we decompose F into its eigenvalues:

\det(F - \lambda I) = \begin{vmatrix} 1 - \lambda & 1 \\ 1 & 1- \lambda \\ \end{vmatrix} = 0

 \lambda^2 - 2\lambda - 1 = 0

 \lambda_1 = \frac{1 + \sqrt{5}}{2}, \lambda_2 = \frac{1 - \sqrt{5}}{2}

 \lambda_1 = 1.61803, \lambda_2 = -0.61803

trace(F) = 1 = \sum{\lambda_i}

det(F) = -1 = \prod{\lambda_i}

We can go on to discover eigenvectors x_1 and x_2. We can then express the Fibonnaci matrix F as

F = \lambda_1x_1 + \lambda_2 x_2

F^k = \lambda_1^k x_1 + \lambda_2^k x_2

As k goes to infinity, the second term goes to zero. Thus, the ratio is dominated by the larger eigenvalue, 1.61803.

Mathematicians in the audience will recognize this number as the golden ratio.

eigenvectors-fibonacci-and-golden-ratio

We have long known that the ratio of successive Fibonnaci numbers converges to 1.61803. Eigenvalues provide a mechanism to derive this value analytically.

Until next time.