If I wasn’t doing research as a physicist, the next best thing would be to build machines able perceive space and time. Few of us think about what our brains are actually doing moment by moment, but if you inspect the activity carefully, you’ll find that over 50% of our cortex is dedicated to visual processing alone. This includes creating a sense of space, identifying objects, seeing faces, analyzing emotional expressions, creating a sense of motion from moving objects, and many other things. It’s very involved. To quote from a book of mine,
“… the parts of the brain that we know have something to do with vision are shown in colour. Two points can be noted. First, someone had to do a lot of colouring — over 50% of the cortex is involved in visual processing. Recently we have begun to be able to chart what areas of the human brain might be involved in visual processing. You might imagine that it would be far less — after all, we humans presumably need lots of brain for doing the things we think we do so well: reasoning, playing chess, doing hard sums, and so on. However, it appears that the amount of cortex we devote to vision is just as large as in this monkey — over half of your cortex is devoted to merely seeing. If we do some very rough sums and generously give hearing 10% of the cortex, all your other senses another 10%, and probably should allow 10-20% for moving all the bits of your body, you can see that there is hardly anything left over for doing the hard bits like chess, cross-words, and perception assignments. But that is exactly the point. Vision is easy because we have lots of brain devoted to it; chess is hard because we don’t have lots of brain devoted to it. If the situation were reversed and over 50% of your cortex were devoted to chess you would, no doubt, be able to take on the most sophisticated computer in the world and beat it. However, computers can now beat the most brilliant of humans when it comes to chess, but no computer in the world can put up the slightest of challenges when it comes to vision. Indeed, we would happily back a humble housefly against the most sophisticated computer in ‘vision Olympics’.”
I oftentimes reflect on this fact when I’m doing theoretical physics and find the calculations difficult. Most of my brain is sitting there idle while a small portion of my cortex deals with symbol manipulation and logic, working tirelessly to follow all these tedious rules I’ve learned over years of schooling. If I deeply understand what the equations mean (which is difficult), I can try to visualize situations with my visual system and that gives me a much deeper understanding of what’s going on. However, you have to realize that these visual areas have evolved here on Earth to avoid predators and find food, so they have a certain way of processing visual information and a very primitive model of space and time. These systems are only accurate for medium sized systems moving at moderate speeds. Matter at the atomic level does not behave in a way our brains easily understand, nor does the cosmos at large. The thought of space and time bending around a black hole or subatomic particles being in two positions at once is almost nonsensical.
We’re just now coming to a level of technology where we can emulate the brain’s ability to perceive 3D space from a stream of images. I doubt the brain creates our sense of space in the exact same way, but I don’t know for sure. As for artificial intelligence, the algorithm is called structure from motion and it’s one of the most amazing things we’ve ever done.
We have two eyeballs which are constantly zig-zagging this way and that way, sending our brain a stream of images from different perspectives. Then nearly three fourths of your brain fires up, over thirty different processing centers, each doing their own task, analyzing corners, finding statistical patterns within the images, finding human faces, identifying objects, sorting them into positions within a spatial environment, calculating your bodily motions to move you across the kitchen, and all that good stuff.
Take motion for instance. We have cells in our brain (the MT area of visual cortex) which are all tuned to motion in different directions. They analyze that stream of images and compare the before and after. If a certain pattern moves from here to there, they fire. If a pattern moves up, another set of cells fire. If it moves to the right, a different set of cells fire. All of these little motion detectors combined give us a sense of moving objects in space. But this system isn’t perfect and you can create weird visual illusions in yourself. Take this Youtube video. Just watch the swirls for around a minute. This will highly stimulate your motion sensors in all different directions. Then the movie cuts to a still image of a cloudy sky yet it will be swirling back and forth for a bit. That’s because your motion sensors are still firing and haven’t settled down yet.
I keep a small library of medical neuroscience textbooks here in my study mainly for my studies in consciousness, space, and time. All of these different processing centers we’ve mentioned can be damaged or injured in a stroke. There are people unable to perceive motion. The medical term is akinetopsia. The world appears as a frozen, unmoving unity. Even stranger, these people can still perceive the world. They have a sense of being in space, there are objects in the room, and they can identify them. It’s just nothing moves. Those motion sensors aren’t working. I’m interested in this phenomenon because I entertain the idea that our experience of time is an illusion.
Other people have brain damage in the areas which are doing things like structure from motion and they don’t have any sense of space. Up, down, forward, backward, left, right, they don’t understand. Some have a sense of space from other senses, such as that of touch, but some don’t even have that. They often can’t grasp objects well either. Their mind can’t do the calculations to orient their hands, arms, and other limbs in the right positions to make certain outcomes happen.
Even still, they can identify things. You can hold a gallon of milk in front of them and they’ll say, “Oh, that’s a gallon of milk.” They just don’t know where that object is in space.
Our brains create a model of the world but this model is very rough and limited. A lot of what we perceive about the world is what we expect to perceive. We focus on what’s most relevant to the task at hand and ignore most sensory data unless we have a good reason to pay attention to it. Take this experiment. If there’s even a momentary distraction, you can switch out entire people, change their gender, their looks, basically everything, and roughly half of people won’t even notice.
And even if you’re paying close attention, there has to be a certain intensity to the change for us to notice it. If change happens very slowly and gradually, we don’t notice it. This is interesting to me because it has to do with our sense of time and the experience of change. It’s weird that in this next video things will be changing, but so slowly your brain won’t notice it. It’ll appear to be a static image, but it’s not.
There is a sharp disconnect between the real world and how we actually experience things in our head. We don’t perceive the real world. We perceive a representation of it, a re-presentation, a second presentation of it, based on a rough inaccurate model with extremely limited information.
You may think the solution to this dilemma is to create better and better brains which can perceive the world with ever greater accuracy, but there’s a problem. If we do that, more and more of the world’s atoms have to be allocated to your brain and its processes. You have to ignore less and less sensory information and need more and more hardware to process and store it all. In the extreme limit, all of the atoms of the universe would go into making your brain and nothing would be left over for the external universe or other people. You and the universe would become one and the same thing, with no external world to be a part of.
If we’re to have an external world where conscious, sentient beings interact and communicate with one another, our awareness of the external reality in which we’re all a part must be limited. There’s no way around it.