Part Of: Machine Learning sequence
Content Summary: 900 words, 9 min read

ML is tribal, not monolithic

Research in artificial intelligence (AI) and machine learning (ML) has been going on for decades. Indeed, the textbook Artificial Intelligence: A Modern Approach reveals a dizzying variety of learning algorithms and inference schemes. How can we make sense of all the technologies on offer?

As argued in Domingos’ book The Master Algorithm, the discipline is not monolithic. Instead, five tribes have progressed relatively independently. What are these tribes?

1. Symbolists use formal systems. They are influenced by computer science, linguistics, and analytic philosophy.
2. Connectionists use neural networks. They are influenced by neuroscience.
3. Bayesians use probabilistic inference. They are influenced by statistics.
4. Evolutionaries are interested in evolving structure. They are influenced by biology.
5. Analogizers are interested in mapping to new situations. They are influenced by psychology.

Expert readers may better recognize these tribes by their signature technologies:

• Symbolists use decision trees, production rule systems, and inductive logic programming.
• Connectionists rely on deep learning technologies, including RNN, CNN, and deep reinforcement learning.
• Bayesians use Hidden Markov Models, graphical models, and causal inference.
• Evolutionaries use genetic algorithms, evolutionary programming, and evolutionary game theory.
• Analogizers use k-nearest neighbor, and support vector machines.

In fact, my blog can be meaningfully organized under this research landscape.

• I explore Symbolist research in my Logic and Algebra sequences.
• I explore Connectionist research in my Reinforcement Learning sequence.
• I explore Bayesian research in my Probability Theory, Bayesian Statistics, Graphical Models, and Causal Inference sequences.
• I explore Evolutionary research in Game Theory sequence.
• I have not yet explored Analogizer research.

History of Influence

Here are some historical highlights in the development of artificial intelligence.

Symbolist highlights:

• 1950: Alan Turing proposes the Turing Test in Computing Machinery & Intelligence.
• 1974-80: Frame problem & combinatorial explosion caused First AI Winter.
• 1980: Expert systems & production rules re-animate the field. 
• 1987-93: Expert systems too fragile & expensive, causing the Second AI Winter.
• 1997: Deep Blue defeated reigning chess world champion Gary Kasparov.

Connectionist highlights:

• 1957: Perceptron invented by Frank Rosenblatt.
• 1968: Minsky and Papert publish the book Perceptrons, criticizing single-layer perceptrons. This puts the entire field to sleep, until..
• 1986: Backpropagation invented, and connectionist research restarts.
• 2006: Hinton et al publish A fast learning algorithm for deep belief nets, which rejuvinates interest in Deep Learning.
• 2017: AlphaGo defeats reigning Go world champion, using DRL.

Bayesian highlights:

• 1953: Monte Carlo Markov Chain (MCMC) invented. Bayesian inference finally becomes tractable on real problems.
• 1968: Hidden Markov Model (HMM) invented.
• 1988: Judea Pearl authors Probabilistic Reasoning in Intelligent Systems, and creates the discipline of probabilistic graphical models (PGMs).
• 2000: Judea Pearl authors Causality: Models, Reasoning, and Inference, and creates the discipline of causal inference on PGMs.

Evolutionary highlights

• 1975: Holland invents genetic algorithms.

Analogizer highlights

• 1968: k-nearest neighbor algorithm increases in popularity.
• 1979: Douglas Hofstadter publishes Godel, Escher, Bach.
• 1992: support vector machines (SVMs) invented.

We can summarize this information visually, by creating an AI version of the Histomap:

These data are my own impression of AI history. It would be interesting to replace it with real funding & paper volume data.

Efforts Towards Unification

Will there be more or fewer tribes, twenty years from now? And which sociological outcome is best for AI research overall?

Theory pluralism and cognitive diversity are underappreciated assets to the sciences. But scientific progress is often heralded by unification. Unification comes in two flavors:

• Reduction: identifying isomorphisms between competing languages,
• Generalization: creating a supertheory that yields antecedents as special cases.

Perhaps AI progress will mirror revolutions in physics, like when Maxwell unified theories of electricity and magnetism.

Symbolists, Connectionists, and Bayesians suffer from a lack of stability, generality, and creativity, respectively. But one tribe’s weakness is another tribe’s strength. This is a big reason why unification seem worthwhile.

What’s more, our tribes possesses “killer apps” that other tribes would benefit from. For example, only Bayesians are able to do causal inference. Learning causal relations in logical structure, or in neural networks, are important unsolved problems. Similarly, only Connectionists are able to explain modularity (function localization). Symbolist and Bayesian tribes are more normative than Connectionism, which makes their technologies tend towards (overly?) universal mechanisms.

Symbolic vs Subsymbolic

You’ve heard of the symbolic-subsymbolic debate? It’s about reconciling Symbolist and Connectionist interpretations of neuroscience. But some (e.g., [M01]) claim that both theories might be correct, but at different levels of abstraction. Marr [M82] once outlined a hierarchy of explanation, as follows:

• Computational: what is the structure of the task, and viable solutions?
• Algorithmic: what procedure should be carried out, in producing a solution?
• Implementation: what biological mechanism in the brain performs the work?

One theory, supported by [FP98] is that Symbolist architectures (e.g., ACT-R) may be valid explanations, but somehow “carried out” by Connectionist algorithms & representations.

I have put forward my own theory, that Symbolist representations are properties of the Algorithmic Mind; whereas Connectionism is more relevant in the Autonomic Mind.

This distinction may help us make sense for why [D15] proposes Markov Logic Networks (MLN) as a bridge between Symbolist logic and Bayesian graphical models. He is seeking to generalize these technologies into a single construct; in the hopes that he can later find a reduction of MLN in the Connectionist paradigm. Time will tell.

Takeaways

Today we discussed five tribes within ML research: Symbolists, Connectionists, Bayesians, Evolutionaries, and Analogists. Each tribe has different strengths, technologies, and developmental trajectory. These categories help to parse technical disputes, and locate promising research vectors.

The most significant problem facing ML research today is, how do we unify these tribes?

References

• [D15] Domingos (2015). The Master Algorithm
• [M01] Marcus (2001). The Algebraic Mind
• [M82] Marr (1982). Vision
• [FP98] Fodor & Pylyshyn (1998). Connectionism and cognitive architecture: A critical analysis