On Intelligence is a book about the brain written by the man whose high-tech innovations fueled the success of the Palm Pilot and other handheld computers and who now aspires to change the fundamental nature of computing itself. Jeff Hawkins’s viewpoint is that of a computer designer—or, as he may prefer more grandly, that of a computer architect—but he seeks his inspiration in a model of brain function that he hopes may spawn a new generation of computers and a new industry. I think his vision succeeds to a surprising degree.
SEARCHING THE MEMORY BANKS
It is in the book’s inner and more meaty chapters that Hawkins and his coauthor, New York Times science writer Sandra Blakeslee, advance their core thesis, which concerns speciﬁcally that part of the brain, the neocortex, that is the substrate of thought, memory, and decision, and therefore, the basis of human civilization past, present, and future. The authors imagine the neocortex as a vast memory bank in which experiences are converted to memories by a simple physiological device known as Hebbian learning. Named for Canadian psychologist Donald Hebb, this well-known explanation of learning claims that connections in the brain that are used repeatedly become increasingly more effective, thereby establishing patterns of preferential pathways for future use.
This vast memory bank is continually bombarded by new inputs, impressions of the world transmitted through our sense organs. Memory operates to ignore all those inputs that are expected, and so lead to predictable conclusions, because they simply conﬁrm that the world remains unchanged. Instead, memory pays attention to those impressions that are unexpected and, therefore, signal that something has occurred that would not have been predicted from previous experiences.
In Hawkins’s model, which draws its inspiration from the well-understood physiology of the visual cortical areas, the neocortical memory is arranged in a hierarchical pattern. Memories based on primary sensory inputs such as sight, sound, and touch are at the lowest level. Levels further up in the hierarchy integrate these primary inputs until we ﬁnd a level that embodies concepts. Thus, our concept of a dog is massive and robust, integrating our entire experience of dogs of all kinds in all situations in which we have encountered them. Hawkins labels this concept an “invariant representation.”
We can detect a dog by its visual appearance, the feel of its fur, the sound of its bark, or other sensations, and, if a dog enters the room, we become aware of it by a change in one or more of our sensory inputs. The detected change is passed to the next hierarchical level of the neocortical memory bank, which compares the impressions coming from our senses with our concept (our invariant representation) of a dog, and decides whether or not this is a dog. If the new sensory inputs fail to match our concept of a dog, the matter is passed to higher levels to ascertain whether or not the inputs ﬁt any other concept or we are in the presence of genuine novelty, such as a new species.
But if the conclusion is that this is a dog, and if we had expected that a dog might appear (because we are in a room with the door open and a dog is roaming around the house), then the process of passing to higher levels ceases. The appearance of a dog has no further signiﬁcance; it was predictable. If it was not predictable, though, the event is passed up the hierarchy to a level that matches the appearance of a dog to other sensory inputs we have stored. Then, against this broader picture of the world, the brain searches around for memories that can explain why a dog should have appeared, and whether it is something that is so unpredictable (because we are in an operating theater scrubbed and gowned and assisting in open heart surgery) that it demands further attention, or whether there is some reason it could have been expected or predicted, and therefore, can be ignored.
According to this theory, as I interpret it, conscious awareness of an event will be invoked only when the sensory inputs cannot have been predicted, and, therefore, cannot be ignored. They require a response. Thus, if Alice had not noticed that the white rabbit wore a waistcoat and took a watch out of his pocket, she would have gone to sleep, and we would have had no tale of Wonderland. The phenomenon of conscious awareness presumably signals the highest level of searching the memory banks.
FROM THE PREDICTABLE TO THE NOVEL
Can the model of hierarchical memories and predictions accommodate the concepts of imagination or creativity? After all, these are some of the most speciﬁcally human traits: to see what has never been seen, to hear what has never been heard, and to imagine what has never existed. Can they arise from a brain made up of a bank of hierarchical memories, leading to successively higher levels of invariant representations? Put that way, these traits cannot. But I believe they can, and Hawkins’s model, if we tweak it a little, allows it. You merely have to add that the brain is not static; instead, it churns things around, endlessly recombining the invariant representations, like a card player shufﬂing the standard deck of 52 cards. A lifelong bridge addict once told me that, however many times he could recall the pack being shufﬂed, he could not remember all the same cards in exactly the same order ever having been dealt twice. In the nonstatic brain, shufﬂing of invariant representations into new conﬁgurations can lead to entirely novel concepts, and these concepts can lead to action.
When an artist paints a scene or portrait in a way never before seen, he is combining elements that have been seen in a way that produces something that has never been seen. A musician writing an entirely new piece is still composing it from notes that already exist. Imagination cannot conceive what has never been experienced, it simply takes things that have been experienced and combines them in new ways. Shakespeare wrote unique things, but he wrote them, for the most part, with words his audience already knew.
The concepts of man and moon have been combined in the human imagination for as long as records existed. Imagination put them together as “the man in the moon,” although they had never been together because, for the ﬁrst 99.999 percent of human history, no man had ever been or could have been anywhere near the surface of the moon. By the time that scientiﬁc capability arose, the concepts man and moon had long been combined in the human mind.
THE HIPPOCAMPUS AS CPU
The brain is frequently thought of as a computer in the head. As an expert familiar with the details of computer design, Hawkins is at pains to explain why his model of the brain is not how a computer works. No doubt, looked at in the detail that a computer architect commands, Hawkins’s model of the brain is quite different from any computer that has been made. But this reviewer, who has worked all his life on the brain, is reluctant to dismiss the similarities. A computer has memory, but it also has a central processing unit (CPU), which is not itself used to store memories, but somehow, in a way incomprehensible to all but computer architects, grinds up information and trusses it up into a format that the memory banks can handle. Hawkins argues that the brain has no equivalent of a CPU. But perhaps he has too quickly dismissed the possibilities inherent in the hippocampus.
The brain is frequently thought of as a computer in the head. As an expert familiar with the details of computer design, Hawkins is at pains to explain why his model of the brain is not how a computer works.
Hawkins has accepted a neuroscientiﬁcally fashionable theory that the hippocampus, in effect, sits on top of the neocortex. By this, Hawkins does not mean that the hippocampus represents the highest conceptual level of the neocortex or is physically on top of it. Rather, he suggests that sensory inputs that are truly novel, and which, therefore, cannot be assimilated to any existing concepts, are dumped into the hippocampus to be temporarily stored before being assigned to permanent homes in the neocortex. Patients with damage to the hippocampus show an inability to remember anything occurring after the injury. Everything stored before the injury is intact. The keyboard has been destroyed, no new ﬁles can be written, but all existing ﬁles are intact and accessible.
I tend to think that the hippocampus, given its function in recording memories, is a CPU involved on the input side. It holds memories only temporarily, up to no more than an hour, before passing them on to the neocortex for permanent storage. It has a simpler structure than the neocortex and a far more streamlined one in an anatomical sense. I am inclined to think that at least one function of the hippocampus can be compared to that of a keyboard: It provides a standard algorithm to process sensory inputs so as to prepare (“format” or “type”) them in the way needed for their entry into the stored ﬁles held in the permanent neocortical memory banks.
On Intelligence makes little reference to sleep and dreams, but they are essential for all vertebrate brains, and for survival. Without sleep, memory fails, emotional stability is lost, and the individual rapidly becomes totally incompetent. In nuts-andbolts terms, the requirement for sleep indicates that the neocortex needs some quiet time, withdrawn from the hurly-burly of the outer world, shielded from inputs and the need to make decisions about action. During this quiet time, the neocortex somehow ruminates over the inputs it has been provided, churns these inputs this way and that as we see in dreams, digests them, and then stacks them away neatly in the memory banks so that we wake up refreshed and with renewed ability to take in and deal with new inputs. The need for sleep shows that the neocortex must have time for some kind of housekeeping function. The reciprocal behavior of the hippocampus and neocortex in waking and sleeping favors the idea that the hippocampus plays a crucial role in this housekeeping process.
The more memories that have been added to a concept, the less change will be impressed on that concept by each new input. But if we conceive the neocortex as a memory bank, there is no reason to think that any new inputs will be discarded.I take exception to Hawkins’s use of the term “invariant representations.” The idea is that the concept of a dog is built from many remembered inputs of many different features of dogs. When some of these many features are detected in new sensory input, such as the sound of a bark or the padding of familiar paws, the concept of a dog is invoked (Hawkins would say “predicted”) in the hierarchical memory banks. So far so good, but, by deﬁnition, not invariant. The concept we have built is simply less variant than the sensory inputs of the latest moment. If the supposedly invariant representation is built from previous inputs, why will it not continue to be modiﬁed, in Hebbian fashion, by new inputs? If, on a visit to Mongolia, we encounter a new breed of dog that can whistle the “Stars and Stripes,” we will at ﬁrst be mightily surprised. But, after ﬁnding that the streets of Ulan Bator are positively infested with such dogs, we will gradually incorporate this information into our representation of a dog and ﬁnally think no more of it. In other words, because the memories are continually being added to, the lack of varying of any representation is relative. The more memories that have been added to a concept, the less change will be impressed on that concept by each new input. But if we conceive the neocortex as a memory bank, there is no reason to think that any new inputs will be discarded.
AGING AND LEARNING
This view of concept formation has an important bearing on a widely held, but not well-substantiated, theory about learning. We know that children learn language faster than adults. The fashionable explanation is that this results from some signiﬁcant change in the developing brain, so that beyond a certain, young age, the brain will no longer accept new inputs as readily as it once did. The problem is that no one has identiﬁed any changing brain mechanism that would explain this supposed age-related deterioration. For example, there is no evidence to suggest that Hebbian learning decreases with age.
An alternative interpretation would be that the concepts in the adult memory banks are derived from the integration of so many previous inputs that new inputs have increasingly less effect. If a baby’s brain is a tabula rasa—a virgin ﬁeld of new fallen snow—then a single footprint will stand out as clear as the crater of a volcano. But once the ﬁeld is tracked with innumerable muddy footprints, it will take a landslide to make a noticeable impression. Such a view of reduced learning ability with age does not require any mysterious change to take place some time between childhood and adulthood. Nor does it imply that a baby is better at learning than an adult. On the contrary, it proposes that the adult brain is wiser, because it is less inﬂuenced by single events and has an incomparably larger and more powerful memory bank with which to assess and deal with new inputs.
Consider a baby seeing a spade and hearing the word “spade” for the ﬁrst time. The baby has no choice but to associate the word with that speciﬁc spade. But to the adult, “spade” can summon up the top suit at bridge, gardening, building sites, funerals, calling for plain speech, or necessary basic hard work—each of these with many associations, great and small, real and imaginary. Now, to ask the adult brain, which incorporates all these multiple layers of meanings and associations, to select the right one and link the English word spade to the word for spade in Chinese or Hindi is a much larger task than anything facing the baby watching his daddy for the ﬁrst time digging in the backyard ﬂower bed. Such an alternative view is, at least, comforting to those of older age.
OUR GRACIOUS SILICON VALLEY HOST
To most people, both computer architects and neuroscientists speak utterly incomprehensible languages, giving even familiar words meanings quite alien to the concepts those words convey to the brains of the rest of the human race. In fact, computer architects and neuroscientists are also pretty much incomprehensible to each other. It amused me that the computer term “memory” refers to that part of the computer that cannot remember—the part that loses all memory when you switch it off—whereas the part of the computer system that does remember is often called a media, which for most people would signify the louche mechanism by which politicians get people to vote for them or the not-unrelated concept of a charlatan’s assistant invoking ghostly spirits.
So it is both refreshing and amazing that Jeff Hawkins has so mastered the neuro- scientiﬁc background of brain structure and function to put forward an exciting, cogent, and stimulating theory of how the neocortex works, even down to the nuts-and-bolts level of how it is wired. Few neuroscientists have attempted such a task. This is an example of how an outsider, coming in with a burning enthusiasm and fresh ideas from a different ﬁeld of specialization, can sweep away cobwebs and conceive a new and important conceptual base for understanding the brain. Neuroscience beneﬁts greatly from such a venture, and if, as Hawkins intends, the beneﬁt also opens up new ideas for future computer architects to develop a model of a hierarchical memory, then On Intelligence is a good thing.
On Intelligence is written in a warm, personal style. We are welcomed into Hawkins’s life and his home. We must overlook the odd lapses into Californian triumphalism, such as commercial success = Truth, or the irritating computerese habit of re-designating familiar words such as “name” and “sequence” or even “intelligence” to mean quite different things from those we would ﬁnd in a dictionary. But we would be impolite guests indeed to raise our eyebrows as Hawkins invites us to share in the hopes and dreams of his early career, and his driving ambition and plans for the future.
From On Intelligence by Jeff Hawkins with Sandra Blakeslee. © 2004 by Jeff Hawkins and Sandra Blakeslee. Reprinted with permission of Henry Holt and Company, New York.
Let’s turn our attention to the largest technical challenge we will face when building intelligent machines, creating the memory. To build intelligent machines, we will need to construct large memory systems that are hierarchically organized and that work like the cortex. We will confront challenges with capacity and connectivity.
Capacity is the first issue. Let’s say the cortex has 32 trillion synapses. If we represented each synapse using only two bits (giving us four possible values per synapse) and each byte has eight bits (so one byte could represent four synapses), then we would need roughly 8 trillion bytes of memory. A hard drive on a personal computer today has 100 billion bytes, so we would need about eighty of today’s hard drives to have the same amount of memory as a human cortex. (Don’t worry about the exact numbers because they are all rough guesses.) The point is, this amount of memory is definitely buildable in the lab. We aren’t off by a factor of a thousand, but it is also not the kind of machine you could put in your pocket or build into your toaster. What is important is that the amount of memory required is not out of the question, whereas only ten years ago it would have been. Helping us is the fact that we don’t have to recreate the entire human cortex. Much less may suffice for many applications.
Our intelligent machines will need lots of memory. We will probably start building them using hard drives or optical disks, but eventually we will want to build them out of silicon as well. Silicon chips are small, low power, and rugged. And it is only a matter of time before silicon memory chips could be made with enough capacity to build intelligent machines...
The second problem we have to overcome is connectivity. Real brains have large amounts of subcortical white matter. As we noted earlier, the white matter is made up of the millions of axons streaming this way and that just beneath the thin cortical sheet, connecting the different regions of the cortical hierarchy with each other. An individual cell in the cortex may connect to five or ten thousand other cells. This kind of massively parallel wiring is difficult or impossible to implement using traditional silicon manufacturing techniques. Silicon chips are made by depositing a few layers of metal, each separated by a layer of insulation. (This process has nothing to do with the layers in the cortex.) The layers of metal contain the “wires” of the chip, and because wires can’t cross within a layer, the total number of wired connections is limited. This is not going to work for brain-like memory systems, where millions of connections are necessary. Silicon chips and white matter are not very compatible.
A lot of engineering and experimentation will be necessary to solve this problem, but we know the basics of how it will be solved. Electrical wires send signals much more quickly than the axons of neurons. A single wire on a chip can be shared, and therefore used for many different connections, whereas in the brain each axon belongs to just one neuron....
Real brains have dedicated axons between all cells that talk to each other, but we can build intelligent machines to be more like the telephone system, sharing connections. Believe it or not, some scientists have been thinking about how to solve the brain chip connectivity problem for many years. Even though the operation of the cortex remained a mystery, researchers knew that we would someday unravel the puzzle, and then we would have to face the issue of connectivity. We don’t need to review the different approaches here. Suffice it to say that connectivity might be the biggest technical obstacle we face in building intelligent machines but we should be able to handle it.
Once the technical challenges are met, there are no fundamental problems that prevent us from building genuinely intelligent systems. Yes, there are lots of issues that will need to be addressed to make these systems small, low cost, and low power, but nothing is standing in our way. It took fifty years to go from room-size computers to ones that fit in your pocket. But because we are starting from an advanced technological position, the same transition for intelligent machines should go much faster.