Skip to content

liquid thoughts Posts

Document ViewSpec Augmented Capabilities

Document ViewSpec Augmented Capabilities#

According to Doug Engelbart, and thus to our current project, we need to augment our collective capability to approach urgent, complex problems in order to gain more rapid and better comprehension to result in speedier and better solutions.  

To increase the speed and quality of our comprehension is an issue of how we can interact with the relevant information we have at hand and to access and interact with the information which is useful and not immediately at hand or even known. 

When reading documents in order to understand an issue it is not practical to expect the knowledge worker to have the time to read every word in ever possibly relevant document to evaluate their relevance and to extract the specifically useful information, including both specifically sought information and that which is relevant but the reader was not aware of.  

Meaning in Symbols#

What the knowledge worker is dealing with is meaning encoded in symbols. Relevant steps when decoding the symbols, of which we are primarily dealing with text, include trying to extract what the author actually meant and what the author’s paradigm was, the coherence of the thought process and the accuracy of citations and references to data.  

There are many relevant symbol spaces (for a more in-depth look at this please visit my site, including social media, knowledge graphs, paper books and so on, and how to connect within and between them will be a major issue in making advanced viewSpecs possible, but that is outside of the scope of this article. Here is a basic laundry list of some potentially useful ViewSpec interactions though: 

Visual Symbol Interaction#

Providing different ViewSpec capabilities can augment the reader’s interaction with the symbols to provide specific capabilities, including: 

Before opening documents:#

  • Automatic text summarisation to help decide which documents to read further
  • Information about the author to understand the author’s perspective
  • Citation and Link analysis to understand how documents relate
  • When reading documents:#

    Ability to specify what to view in the document, to set a ViewSpec, on the fly, without cognitive load for the trained user, with as little effort as a twist of the hand which a sculptor or painter might make. These view interactions should include: 

  • Specify the depth of the hierarchy to view: headings only, full text or headings with first line of every paragraph etc. to skim the document to get a feel for what sections cover what information
  • Choose to view only sentences with a specific keyword in order to see how that specific keyword is covered. Also useful for when reading and forgetting what a term was and quickly being able to see the first time it was used/introduced in the document
  • Setting how the text should appear, such as colour coding text based on a glossary and specifying aspects of the typography and layout to the readers preferences
  • Follow explicit links which the author has set, in a manner which allows the reader to efficiently get further information (ie. links to specifically relevant sections, not just to whole documents or books or whole videos but to specific paragraphs and to specific moments in videos) and to judge the veracity of the information as presented
  • Follow implicit links based on the text to search the web, references and more
  • Using glossaries to allow the author to elaborate on specific terms with some effort and quality without the need to do it every time the term is used
  • Using ‘hyper’-glossaries to connect to other information spaces, such as transitioning between documents and knowledge graphs
  • When authoring #

    In this model the author has a clear obligation to make published material as useful and accessible to the reader, not only the reader within the very specific domain but also the more general reader, since the goal is the Concurrent Development, Integration, and Application of Knowledge, not simply to publish a document for the sake or it.  

    Authoring to support the reading process by: 

  • Most clearly presenting the author’s intention and making it as verifiable as possible 
  • Provide multiple levels of access for readers, such as provided through approaches to glossaries and glossary-view mechanisms
  • During the authorship organise the information the author has access to and has produced, in a flexible way to help the author then by showing relationships, patterns and to determine what is relevant the understanding as well as preserving these views for readers
  • Publish (make public) the information/document in technical formats which makes them as accessible to systems to analyse and present to the reader as possible, through different reader perspectives and over different time periods (integrated archiving)

Leave a Comment

Power Tools for the mind

I am writing this since there are many who maintain that when it comes to intellectual work ‘it’s not about the tools, it’s about the human side’ and that ‘what the world needs now is not different types of links’. This is in the shadow of discussions of the greatest tech demo the world has ever seen. On the 9th of December 1968 Doug Engelbart did not stand on a stage and talk about his strategic vision or the philosophy behind the work – vitally important as though that is. He didn’t tell us anything. He showed us. He demonstrated. It was a ‘demo’ – not a sermon nor an outline of how we should work in an ideal world.  

The article makes the point that technology matters and useful tool systems do not just ‘happen’ they are developed for reasons and we as a community need to consider what reasons we feel are important.  

History – Tools Deliver Capabilities 

There were no days when the only tools we needed to build a dwelling was our own hands to break some twine and fashion a cover above us. Homo sapiens come from a long line of tool users, we have always been tool users. There were definitely days where the tools we used were rudimentary and direct – a simple hammer striking a nail of some sort and into the wood it would settle.  

The power and control the tools provide determines the power and control we have over what we want to build. We could not have built a skyscraper with hammer and nails and certainly not a high speed car, train or aeroplane with any tools from an earlier era – we are always making to the tools maximum potential. If we want to go beyond what we can make today, we need to improvement our tools. 

It is the same with mental tools. We could not build anything beyond the most rudimentary without tools to help us plan and even to conceive of the project. This highlights the need for powerful Tools for Thought (Rheingold, 2000), as Howard Rheingold put it.  

Future – We are at the Start, not End, of Digital History  

That was the past and it’s something we can read about and some us even lived there. But what about the great un-seen future? There are several vectors which seem quite safe to assume will continue, such as faster computers and faster networks, something which we have already reached a level of speed where promises of improvements in the future makes even some techies yawn. Some of the capabilities they will support also have safe future improvements to consider, including VR and AR. The immersion we can already experience with a VR headset or a more sophisticated headset from and of the PC platform suppliers is surprisingly deep. These are just the very earliest days of this display and creation/capture technologies though. Next time you move home try to get fibre to the home and you’ll be met with questions of why you would need it and reply that you expect you will have a few 8k per eye headsets lying around and high fidelity audio, HDR 8K cinema screens in a few years so please, deliver us faster broadband! 

Picture the future of AI and CGI in areas of entertainment. Now have a coffee or go for a walk and think of similar improvements to the systems and tools we use for knowledge work   :-) 

What the tools should augment – Our Mental Space 

We are working to augment the human mind to better approach complex problems. The future of video games becoming ever more real and hyper-real is assured, as is the future of passive moving entertainment – the markets will take care of this since the instant value of end-user investment in these entertainments is self-evident enough for the end-users to finance this themselves. This post concerns the future of human thought and communication, something few feel they have either ownership over or control over to co-create. First of all, I make the assertion that what we should augment is our mental reach by enlarging our mental space through capacity and rich interactions. 

  • Even though an Einstein can have thought experiments and make ground breaking discoveries by ‘pure’ thought, there is a reason Einstein is famous, he was a very rare person indeed. Nevertheless, he was a person who used tools, including language, mathematical symbols and a blackboard to extend his mind and, like Richard Feynman, the symbols he worked with was not just a record of his thought, but very much a part of his thinking process. In Supersizing the Mind Andy Clark writes that “Feynman was actually thinking on the paper. The loop through pen and paper is part of the physical machinery responsible for the shape of the flow of thoughts and ideas that we take, nonetheless, to be distinctively those of Richard Feynman” (Clark, 2010). 

  • Rich Interaction – The Fundamentals 

    The shift from a passive substrate to an interactive substrate will cause shifts beyond what we can imagine but we are already living with the first shockwaves of this transformation because of the use of various human and software agents influencing elections through (primarily textual but all completely digital) ‘social’ media – Donald Trump’s tweets have fundamentally different characteristics than Martin Luther’s Ninety-five Theses.  

    But let’s go deeper. What is most fundamental in the universe is not matter, nor is it information – it is interaction. This becomes self-evident when we consider that information arises out of interaction.  

    “What I hear, I forget. What I say, I remember. What I do, I understand.” Said Laozi. It is the ‘doing’ that matters and to do more, to reach further, to determine what you seen and how you see, we need to build ever more powerful interactive tools and media. You cannot see further than your telescope allows. You cannot drive longer than your fuel tank allows.  

    We have the opposing concepts in our culture regarding our abilities. On one hand there is the ‘limitless’ potential of the human mind versus the excuse of being ‘only human’. Our basic mental processes vary but aspects of our mental capabilities, such as working memory, have clear limits.  

    Similarly, to reach deeper into the information than your information manipulation tools allow you to do.  

    Our basic human muscles are augmented for locomotion and our basic human mental capabilities are augmented for mental motion, as enabled by our tools and our education.  

    The Primary importance of Addressability 

    To refer back to the introduction; Different kinds of links matter because different kinds of ways to connect knowledge is the very fundament of what knowledge is. and you cannot interact with something you cannot point to or grasp. Doug advocated high resolution addressing and this will change the very flexibility of our mental spaces.  

    We are working on this in the Doug@50 community, particularly how to address from one domain (knowledge graph, traditional document etc.) to another, not just to a ‘document’ or a view specification, but to specific points.  

    Richly Interactive Digital Space  

    Much in the same way the ancient Sumerians needed to understand the characteristics of the clay they pressed their reeds into in order to record their thoughts and the Egyptians needed to understand the characteristics of papyrus, we need to understand the characteristics of our digital workspace and that goes much further than simply trying to mimic the workspaces we know from the pre-digital world. Doug was never fond of the notion of WYSIWYG, referring to it as What You See Is ALL You Get. For use to truly augment our minds with powerful tools, we need to investigate the properties of the digital space with a view to making it ever more interactive to provide us with ever more powerful commands, something I refer to as creating a more Liquid Information environment, which is my philosophy developed with Sarah Walton and which Doug could strongly relate to. 

    In Doug’s language, we need to evolve our paradigms to better understand and develop powerful digital tools:  


    The ’Stuff’ of Interaction – Symbols 

    As far as symbol manipulation goes, I have blogged on this and built a very small web page, where the TL:DR summary is simply: No matter what thought augmentation tech we will build, it will somehow have to allow us to interact with symbols in some manner and the potential there are indeed rich:  

    A few Words about Text 

    Humanity has spent something like five and a half thousand years refining symbol authoring and reading, a system “developed to solve very different problems than speech” (Harari, 2018).  

    I am writing on this topic under the Symbols category. 

    The evolution of the written word has thus far been confined to a passive substrate, that of hardened clay, papyrus and paper. With the interactive computer digitally powered substrate we have now is the opportunity to vastly improve our symbol interaction by, in essence, making Einstein’s blackboard any size and shape he would have wanted (dynamically), while also carrying the entire history of work done on it, linking all aspects of the information it contains and with all other known work. For example.  

    This is the field I am passionate about and what I am passionately defending and promoting in this article. Other work will be crucial, such as knowledge graphs and AI, but is outside of my personal understanding and thus outside of my remit. It bothers me when symbols in the form of text is denigrated to being old fashioned where what I see is huge potential. Doug agreed: 

    “I honestly think that you are the first person I know that is expressing the kind of appreciation for the special role which IT can (no, will) play in reshaping the way we can symbolize basic concepts to elevate further the power that conditioned humans can derive from their genetic sensory, perceptual and cognitive capabilities.” 

    Doug Engelbart in personal email to Frode Hegland September 2003 

    Rich Tool Interaction & Doug’s Goals 

    The goal of rich interactions for the human mind is not for it’s own sake, it is in order to deliver on the aims of augmenting our capability to approach urgent, complex problems in order to gain more rapid and better comprehension to result in speedier and better solutions, as we have summaries Doug’s goals on  

    Please note, the rich interaction is not with the tool – it is through the tool – with information and people. 

    Coevolution of Mind & Tools : Deeper Literacy 

    The term I use for what richer interactions provide is deeper literacy, which I hope encapsulates both the advanced use of powerful tools and the mental (human side, you could say) understanding of the work and thought process which the tool can augment. I use the term ‘deeper’ to highlight the need to not simply work to an end goal of having a ‘deep’ literacy but to continually improve, and use the tools we build to go further, in other words, to bootstrap an ever deeper literacy: 

    It’s not enough to become a master of the tools we use to serve human processes as we understand them today, we need to continually develop the tools to provide further depths and richness, beyond what our imagination can see – learning new opportunities for augmenting thought as we become experts in using different kinds of tools – we should not be limited to what our imaginations tell us we need before we know through experience the current and next stage of powerful tools intimately well – we need to use our tools to bootstrap our understanding of what future tools can give us.  

    The same way we could not expect a medical scribe to usefully add capabilities to Microsoft Word or The Web, we cannot usefully conceive of advanced and useful tools if we don’t use them and constantly continue to develop them. As was (apocryphally) attributed to Henry Ford; if I had asked people what they wanted, they would have said faster horses. 

    If you please have a look at the previous section again, on ‘Rich Interaction & Doug’s Goals’ and consider what’s described there as the goal of the augmentation systems, I hope that it becomes a nice loop of how we need to constantly refer to what the actual goals are and to inform our development based on actual advanced tool use.  

    The Space of Evolution – The Capability Infrastructure 

    It is important to take into account that our capabilities are always part of larger infrastructures. An example is the basic tools of reading a writing; the pencil and paper. This is not one tool nor two; They are produced by manufacturing of wood products and graphite which come from locations far from their use, transported over transportation networks into factories where other components are added, such as paint and the rubber and the metal to fasten the rubber. The pencil and paper then needs to be distributed so that they may reach the consumer who can then use these ‘tools’. This is not even taking into account the education infrastructures to for the user to know how to write.  

    Doug’s Capability Infrastructure Map is comprised of two pillars, which he named the Human side and the Tool side with the Basic Human capabilities underneath, illustrated with many connecting lines between these sections, to highlight their interdependence. What often happens when people want to further his work is that they ignore the Basic Human capabilities underneath and then pick a side to focus on; Human or Tool side. The crucial point is however that the capabilities reside in the connections between the sections as is illustrated in some versions of this document, such as the one at The importance of this cannot be overstated – augmenting our capabilities is inherently what we might call ‘interdisciplinary’.  

    I would contend that the goals for what these capabilities is not inside the Capability Infrastructure. The infrastructure is can be employed to carry out work of any morality (or lack-of). The goals of what we should achieve through the our capabilities need to have their own space for discussion so that the capabilities can be improved to deliver on the goals. My understanding from working and talking with Doug is that this was very much implicit in his work. He himself was a deeply warm and moral human being so my take on it is that he simply didn’t fully see the need to develop a message for this beyond his core ‘augmenting our capabilities to solve urgent, complex problems collectively.’ 

    We ignore the moral direction of our development work at our peril and we should therefore probably devote some dialogue time to what we really feel we should actually work to augment.  

    The Framework of Evolution – Facilitated Evolution  

    Doug discussed the need to facilitate evolution and this is a hugely important aspect to his work, not only as it shows his intellectual openness and lack of pride in his specific ideas (he always expected others to go further and was very disappointed after the 68 demo that the category of work didn’t seem to take off) but because it accepts our lack of understanding of any ‘ultimate’ future end point and highlights the need to ‘fumble’ our way to a better future, learning as we go along.  

    { Please note there have been comments on how evolution is not evolution if it’s facilitate. This is something which can be argued over coffee for those who would like to take the time but please consider that Darwin used the word ‘evolution’ only once in writing and preferred ‘descent with modification’ which makes more sense. The earlier meaning of the word was simply ‘growth to maturity’ which fits the current use of the term well } 

  • Evolution is a product of environment and actors co-evolving, so if someone changes the actors or the environment the evolutionary pressures change.  

I feel this is a very strong concept and speaks to the complexity of the world. We simply cannot ‘design’ everything we need or want because the world is too complex and our needs and wants will change as we evolve ourselves as well.  

{ Another Aside: A real-world instance of facilitated evolution is education, where the curriculum and the culture of the country and the specific institutions provide a framework for thought and behaviour, facilitating innovation in some direction and curating it in other. A curious artifact of this in the United States is how people of a right-wing persuasion say that universities have become ‘infested’ with left wing ideology, not for a minute considering that maybe getting a good education opens and enlarges the mind and this is how a more free-thinking, or ‘liberal’ mindset manifests. } 

The need to somehow facilitating evolution then asks us to look at what the elements of our thinking and working are and how we can frame the space of the innovation. 

This must be a core part of our process.

Leave a Comment

For the Love

The reason for Doug’s success comes from his philosophy but his ‘philo-sophy’ was not simply a love of knowledge, it was a direction for the creation and refinement of human and tool systems. Doug’s initial aim was to improve the human condition. THIS was his stated aim and as a kind a loving character, he did not need to spend much time dwelling on this, it was simply who he was. As a global society though we do not have this luxury. My young son Edgar, who will turn one year old in this 50th anniversary of the demo, has already travelled to Norway, Denmark, Japan and Azerbaijan and he was met with kindness and love everywhere he met (admittedly he is an amazingly beautiful and charming child). All these ‘territories’ have been at conflict with some other territory and have some adversarial conflicts internally to this day.

I believe strongly that how we work to create harmony for all must be a prime driving focus for all the work we do in Doug Engelbart’s name, otherwise we are simply augmenting ignorance as much as enlightenment, hate as much as love, prejudice as much as open-mindedness.

As a teacher and father I feel I must highlight the prime importance of the moral side of the equation and yes, I put my money where my mouth is: I made the point at a conference with over 40 world leaders just last week that this is really about ‘Love’, a surprising comment which my friend the moderator needed to ask me 3 times to repeat since it seemed so out of place.


Leave a Comment