Information Entropy
Animal sounds were the first massive reduction in life-form-based information entropy on this planet. The sounds and the information they carried reduced the relatively limitless possibilities of data that life could generate on Earth. After humans came on the scene, further reductions in information entropy occurred with the introduction of language, written symbols, mathematics, and, much later, the printing press. Each of these developments reduced the average level of information surprise or uncertainty in the possible outcomes of life-form information. As AI has shown, entropy will continue to decrease in the Earth’s infosphere. Every new development has increased efficiency and standardization, with AI doing so spectacularly.
We may even be programmed to do so. Dr. Melvin Vopson has developed the 2nd Law of Infodynamics, which states the opposite of the 2nd Law of Thermodynamics, requiring information entropy to remain constant or decrease over time – which is certainly what we have seen. Further, his theory can predict genetic mutations in organisms, including viruses. TL;DR, the genetic code loses entropy as the mutations occur. The information driver of our individuality requires us to lose information entropy.
As humans introduced technology to organize and distribute information, starting with the printing press, information entropy began an accelerated reduction period. Telecommunications standardized terms while massively expanding the spread of popular languages. Minor languages began to disappear. We learned to persist information for extended periods and with fewer errors, reducing the surprise of new or different information. Later came the markup languages that restrained how information could be displayed. Wikidata standardized knowledge in a way that computers could quickly manipulate and integrate.
And then along comes AI. The simplest way to describe something like GPT-4o is that it is the most extensive, massively organized, lowest information entropy per pound in a singular defined system on this planet by a considerable margin. Other than that, it is just data wrapped in software. If there is no model training or user prompts, guess what GPT-4o does? It just sits there, like what my wife accuses me of doing on Sundays.
What is impressive to me is that we humans keep at this reduction, not that Dr Vopson is correct, which I am sure he is. Besides the 2nd Law of Infodynamics (2LoI), nothing in this Universe requires humans to keep organizing data. Most of us have no idea about the 2LoI, let alone information entropy, yet we all contribute. We paraphrase. We organize our photos and create folders on our hard drives. We use the three acts and the hero’s arc. We Americans always greet with some form of, “How ya doing?” Loony Tunes always ended with, “That’s all folks!” It’s Mac or MS, Android or iOS, but organize our information with something we will.
We need to understand this. My feeble mind says that quantum’s observer and Dr. Einstein’s relativistic one have some role in this information entropy law as the path from sounds to AI is all about observation. We see something, we communicate. We hear something, we communicate. We learn something, we communicate. Now, we know how to persist our observations from hieroglyphs to token embeddings. The act of observation locks in the physical world to us, and thus for any observation and for the 2nd Law of Thermodynamics to hold, the observed entity has to shed some entropy, radiate some heat, or throw off information. The 2LoI is the means to keep systems balanced. Communication is a transfer of energy, from chemical-electric to sound waves and back to chemical-electric. We know the brain radiates heat, but is some of that heat reduced by the chemical storage of information?
For what reason, though? Dr. Vopson has proposed several experiments to illustrate a mass-energy-information equivalence principle. He suggests that the elementary particles store information about themselves just like human DNA. He goes so far as to state, “The mass of a bit of information at room temperature (300K) is 3.19 × 10−38 Kg.” It is tiny but quite possibly measurable. Dr. Vopson’s work may even lead to finding that information and its mass at scale is actually the dark matter we hunt today.
One significant consequence of this line of thinking is that AI is a natural result of information emergence. Today’s AI and its progeny are part of the evolution of the Universe. We discover the laws of nature and apply them for our use. Those laws bind us, so our applications adhere to them, and the information they produce mirrors the constraints of physics. One great example is Zipf’s Law. Basically, some things occur a bunch, while most occur little. Letter, word, sentence, and knowledge domains follow Zipf’s Law. Zipf’s Law is perfectly illustrative of entropy reduction. If every item occurred at the same frequency, entropy would be high, as you would never know what sentence is coming next. Communication would be impossible. AI is not to be feared; it is to be understood, as it belongs in this Universe as much as we do. In fact, the Universe requires the massive entropy reductions that AI provides as we humans observe and persist more and more information, along with the steady Universal increase in entropy. No wonder dark matter represents 85% of all matter.
Our next steps may be some in the path we have already traversed. We are doing amazing things with AI software. However, we know that we have already hit the ceiling. We require an exponential amount of data and compute resources to get to the next level of accuracy. Currently, we shred our communication into anywhere from 384 to 2048 dimensions. What if we just placed our ideas in three? What if we just graphed out how our ideas connect in differing contexts? What would we find? Our research says that about 40% of sentences in Wikipedia are paraphrased concepts. Around the large, repeating concepts are clouds of close semantics, some paraphrases, and some we term free-range sentences. Even knowledge domains follow Zipf’s Law, which makes sense. Way more sentences are expended for the Eras Tour than for the St. Augustine Distillery Tours.
Right now, mathematicians own AI, and I think that is righteous. However, we need more interdisciplinary scientists to look into the smoking gun that is information entropy. Something foundational is happening there, and we are woefully ignorant of how information entropy plays in the Universe.