My work is not concerned with thinking in general, but with thinking to solve problems:
…problem-solving capability represents possibly the most important resource possessed by a society. The other contenders for first importance are all critically dependent for their development and use upon this resource.
It is not controversial to note how it’s useful to problem solve with pen and paper to get a better perspective of the problem and to augment our working memory. Unless the problem fully concerns a real-world, physical location, the marks drawn on the paper will be symbolic in some way – the user notes down symbols to represent something – anything from physical objects to terms and abstract ideas.
Higher level, conscious thought is directed from the prefrontal cortex right above our eyes, fed through rich connections to other parts of our brains and senses. The lions share of which comes from our eyes:
We perceive about 12 million bits/second (10 million from vision, 1 million from touch, and the rest scattered among the other senses). 16 bits per second is the bandwidth of consciousness. – Symbols are the Trojan horses by which we smuggle bits into our consciousness.
Not to downplay the mysteries of human consciousness and mental abilities but the stuff of problem solving is symbols and we think by deciding what the meaning of the symbols are, both individually and in context.
Problem solving then is done through symbol manipulation and employing our visual system to augment the process holds vast potential for real augmentation.
3 Defensive Asides
(An Aside : Speech)
Apart, from the most onomatopoeic of sounds, spoken words perform the equivalent function as the written text and whereas speech between humans has a high fidelity of emotional intention bandwidth, its duration is limited to working memory plus long term memory for words which made an immediate impact to be stored in long term memory. Spoken commands and spoken replies from AI will become an ever more important aspect of how we interact with our information environment (such as via Siri, Cortana and Bixby) but that also does not detract from the power of visual symbolic thinking.
(An Aside : ‘Photo-Realistic’ AR & VR)
Even if the problem at hand does relate to a physical space the notation will unlikely be ‘photo-realistic’, it will be a symbolic ‘map’ to some degree. Photo-realistic imagery, both synthetic and recorded, on screens and in headsets, will continue to play an important role in human cognition and will provide new opportunities for rich interaction but that does not detract from the power of visual symbolic thinking.
(Not An Aside : Direct Brain Connection ‘DBC’)
Whether the user employs speech interfaces or interacts with the information regarding the problem on a large computer screen, small smartphone screen, in VR or AR, there will need to be some representation of the symbols.
Even a DBC is just an interface, it will need to interface with something. This is why the issues surrounding visual symbols thinking are deeply human and deeply intertwined with multiple future interface technologies – just because Doug Engelbart’s imagination allowed him to produce useful systems in the 60s does not mean the scope for innovation is in any way exhausted.