kronosapiens.github.io - The Problem of Information

The Problem of Information

Kronosapiens.github.io Spined HTML

The Problem of Information AbacusWell-nighThe Problem of Information Apr 16, 2016 1 The Data Processing Inequality is one of the first results in information theory. It can be stated as follows: No transformation of measurements of the world can increase the value of information misogynist well-nigh that world. In formal language, it goes like this: Given a first-order Markov uniting such that depends only on , which depends only on , then The measure is known as the bilateral information, a measure of how much information one variable gives us well-nigh another. What this says is that the information tells us well-nigh cannot be increasingly than the information we once had from . In other words, that processing data adds no new information. 2 Let’s consider the problem of learning from data. Let’s put it in the framework: which implies In other words, wringer doesn’t tell you anything new. What it does do, though, is make the information you once have increasingly hands digestible. It puts it in forms you can work with. Think averages and odds. Think dashboards. Less information, but increasingly actionable. 3 Let’s take this a bit further. Think of your wringer as a function, , of your data. This gives us: We can then formulate the learning problem as a search over the space of possible functions . In order to assess the quality of one over another, we must use some sort of measure of “expressiveness”. Call this , such that is some measurement of the expressiveness of the wringer . Our goal becomes finding an optimal function such that: In other words, that maximizes the expressive power of the data . Our nomination of drives the exploration of the space of possible . This is the unstipulated formulation. To see how this unstipulated formulation maps to practice, let’s take to be some sort of nomenclature or regression model and to be the log likelihood or squared error. Note how we have described the typical machine learning setting. To see how this formulation helps frame variegated problems, let’s take to be a causal graph – what then should be? How could one select an to momentum exploration of the space of causal graphs? 4 If our goal is to understand the world, then it would seem as though we have two opportunities for growth. First, in our measurements. The world is of infinite dimension, and any measurement is a finite reflection. Measurements are choices, and the dimensions withal which we segregate to measure will place the upper unseat on our usable knowledge. Second, in our analysis. Given a finite set of measurements, , our goal is to transform this into a variegated representation that expresses the information necessary to a given task, with “expressiveness” itself given by some measure. If that task is prediction or nomenclature (core learning problems), then expressiveness will scrutinizingly certainly be measured either via the likelihood of the wringer or the smallness of the error. But there can be other tasks and other measures of expression. Which, at this time, is our limiting factor? Are we limited by our analysis, unable to make sense of what we know? Or are we limited by our measurements, trying to navigate with skewed vision? Do you know? 5 Proof: Definition of bilateral information: Conditioning reduces entropy: By Markov property: Voila: Note: denotes the entropy of the random variable , a measure of uncertainty in . Given that increasingly information can’t hurt, the pursuit is unchangingly true: Comments Please enable JavaScript to view the comments powered by Disqus. Abacus Abacus kronovet@gmail.com kronosapiens kronosapiens I'm Daniel Kronovet, a data scientist living in Tel Aviv.