Table Of Contents
- Three Interpretations Of Probability
- The Nature Of Noise
- Converting Noise Into Knowledge
- Can All Noise Be Explained?
- Wrapping Up
In this post, we will dip our toes into philosophy of statistics. Specifically, we will explore my experience as an electrical engineer to understand the nature of noise, and chance, in our models of How The World Works. This post is rather tentative, and should be subject to considerable maturation over the coming months (especially once I revisit quantum mechanics). I am releasing it now, because I need to appeal to it in my other work! 🙂
Three Interpretations Of Probability
Probabilistic reasoning is the vehicle to understanding stochastic phenomena. At least three camps exist within the statistical interpretation of probability:
- Objectivists interpret all, or most, probabilities as things that exist in the world.
- Frequentists interpret all, or most, probabilities as things that represent frequencies.
- Bayesians interpret all, or most, probabilities as measures of personal uncertainty.
Consider the different ways in which these three schools may represent the following situation:
Divide some arbitrary surface into four sections, quadrants A, B, C, and D.
Suppose that we know that photon Λ has a 50% chance of being locating in quadrant C.
- Objectivists would claim that Pr(0.5) means that the proton objectively may, or may not be, in that location. This type of stance would, perhaps, leverage an objective interpretation of the quantum wavefunction.
- Frequentists would claim that Pr(0.5) means that, if the scenario happens (say) 1000 times, 500 of these scenarios would see an Λ-C pairing.
- Bayesians would claim that Pr(0.5) means that our subjective valuations of an Λ-C pairings are at 50%. Suppose another observer finds more evidence in support of an Λ-C pairing. For them, but not for us, a Pr(0.75) valuation might be called for.
The Nature Of Noise
My first experience of really wrestling with noise comes from my undergraduate days, while pursuing my electrical engineering degree. I burned a considerable number of hours in front of my oscilloscope, trying to make sense of noise in my circuits. For example, in class we built models of how resistors and capacitors would behave in the presence of one another. From the equations C = dV/dt and R = I*V (themselves derivable from Maxwell’s equations), we built the following prediction of an RC circuit:
However, when you actually build the thing, and measure its waveform, you see imperfections – noise – in your oscilloscope!
Whence these tiny, irregular bumps? It turns out that inferring the origins of noise is hard. In this example, it could stem from:
- Error in our measurement instruments (stray impedances within the oscilloscope circuitry?)
- Inadequate detail in our representation of reality (stray capacitance within our supposedly featureless wires?)
- Violation of our model’s approximating assumptions (the above equations are simplifications of Maxwell’s Equations)
- Failure of our model’s theoretical stratum (historically, Maxwell did not have the last word on electrical phenomena)
- Stochasticity of nature (perhaps spontaneous electromagnetic bursts, e.g., solar activity, are playing a role?)
The takeaway: be suspicious of the words “noise”, “chance”, and “luck”. Entire universes of causes may lurk behind these seemingly straightforward phrases.
Converting Noise Into Knowledge
Not only is the simplicity of “noise” deceptive, but its boundaries change all the time! To illustrate, let me turn to another type of electrical signal, the basis for the binary logic in our computers: the square wave.
Let me now discuss three examples of how noise is this domain eventually found itself within the domain of human understanding.
Example 1. In real-world applications, square wave signals sometimes look like this.
The above measurements seemed like noise… until scientists connected it with the mathematical work of Joseph Fourier. Today, this Gibb’s phenomenon is a known consequence of the summation of multiple frequencies. It is a solved problem. You can even reproduce the phenomenon, and control the height of the “sharp corners” via Java applets. Amazing!
Example 2. In real-world applications, square waves sometimes look like these waveforms.
It is important to remember that such phenomena, subsequently named undershoot and overshoot, were once considered to be noise. However, advances in control system theory, and in particular our models of damping, have explained the above phenomena as well.
Example 3. In the previous two examples, the unexplained noise exhibited patterns. But what about noise that looks a little more like the oscilloscope image I showed above? Can less reproducible noise – noise that looks like the static on analog TV sets – also be made subject to explanation as well?
Yes. Suppose we were to transmit a square wave using an antenna that is a few centimeters wide. Given the spectrum of light, the antenna would be particularly susceptible to microwave signals. What would we see in these transmissions?
Well, it turns out that we would expect to see a certain kind of noise at this frequency. All photons that hit the earth began their lifetime in the past. Some photons hitting the Earth right now began their lifetime 380,000 years after the Big Bang, at the point when the atomic structure of the universe just began to allow light to travel freely. These photons – almost like time capsules from another world – cause some of the microwave-frequency static we see in some of our antennae.
Can All Noise Be Explained?
Let me now embarrass myself, by speaking of things I do not have full command of, yet. 🙂
Perhaps these historical trends cohere most easily with a Bayesian interpretation of statistics: perhaps random noise can be best modeled as a sum of our ignorance. Is this true for all noise? How large is the space of solved noise candidates?
There is a case to be made that objective interpretations of probability are not all wrong. Consider quantum mechanics.
On some interpretations of the wavefunction, including the Copenhagen Interpretation, uncertainty about its momentum and position is embedded into the fabric of spacetime. On this way of thinking, objective uncertainty does exist. The problem of how quantum uncertainty gets “filtered out” to the Newtonian determinism of physics at classical scales is an open problem. On this view, noise will (necessarily, indefinitely) remain a feature of our intellectual landscape.
But other interpretations of the wavefunction, e.g., the Many Worlds Hypothesis, exist. Such interpretations hold that the only uncertainty is that of the observer not knowing which universe she is presently living in. Here, chance or noise is simply a feature of a decision-making apparatus. We are uncertain about which slice of the multiverse we inhabit, and the process of mapping our models to the Real World is computationally expensive.
To close with a complaint: I have not yet encountered a measure for “how much noise remains in my data”. As we saw in the last section, sometimes our models grow as they are fed solved noise; however, I currently possess no way to quantify this progress.
What should you take away from this article? Perhaps the following:
- While the mathematics of probability largely enjoy a scientific consensus, its interpretation is still unclear.
- Noise can come from many sources, potentially including measurement error, model error, and physical processes.
- Not all noise is permanent. Some noise can be solved via scientific innovation.