Skip to content

Category: Symbol

A Clear & Present Opportunity

Time for Better Literature Research

Universities and reading glasses were invented roughly simultaneously (coincidence?), during the 13th century. There were counties for students to have books copied for studying of which I won’t waste our time going through the details of how it wasted their time. It seems however, that not much has changed. Today’s academic is more likely to print out the sources for their literature review than to read it in digital form (Walsh, 2016), even though there are many different types of digital devices to read them on, from reflective to transmissive tablets and from laptops to desktops.

I am not writing about the pros and cons of paper versus digital for general reading but I do mean to highlight that there is a real opportunity to build an ecosystem where digital reading of professional material (someone one reads to learn, not for enjoyment primarily) can benefit from the potentially connected nature of digital text environments.

We will necessarily have to start with PDFs since they are the dominant form of academic knowledge transmission units, though of course support ePubs and other formats as well.


The suggested flow is simply this:


A user reads a document for their research and highlights interesting sections, writes notes in the margin and also generally about the document. The user also also jots down ideas as the reading and research process progresses. Furthermore, the reading applications gives the user a rich set of facilities to view the document as she sees fit, including flexible ways to access keywords, summaries, abstracts, links, references and connections and so on, some of which we have implemented in Liquid Author and Flow.


All the literature review the user has read is accessible through keyword search, including specifying whether to only search highlighted text, annotations, notes or full text. When the user copies text from a document and pastes it into the document their are authoring, all the salient citation information is included and is copied across (since this is not possible by default, they system will need a fast and elegant way for the user to help add this to the document on opening/at leisure and in future automatically use what authoring systems will append as meta-information).

All the literature review the user has yet to read is also accessible through searches and can be bunched based on keywords, citations and other criteria.

All the literature the user has yet to access, which is connected to the users work by being cited in current work is also accessible.

The visual space of connecting literature can be developed by anyone since the connectors will be made clearly available. The weblau is one direction which can be powerfully useful in this regard.

A more Liquid Reader

This could make the literature review process a truly liquid, smooth and rich affair. This is not rocket science or heading far out into the unknown, this can provide clear and immediate benefits.

We have started experimental work around this workflow, with the Liquid | Reader but this will require more investment to continue til completion, particularly since the standards of searching for academic document is so incoherent and therefore many searches engines will need to be strung together.

1 Comment

The Old Man

old man inventor?

As my first major software application reaches 16th most popular Productivity Application on the macOS App Store I reflect again on what Alan Kay said over lunch: “But you CAN do anything!”. I know he was being sincere, it’s the same thing I would say to a curious and passionate student. My concern is that my brain is fully myelinated, all the fatty optimisers have been laid down, my axons and dendrites are not as much in the dating market as before. I have now managed to produce one instance of my vision in Liquid | Author, which includes Liquid | Flow. It’s a different type of word processor where I had a few medium sized ideas (tag for meaning such as for headings, a different Find command and so on) and did a lot of polishing on use, reducing as many button presses and unexpected frustrations as I could see.

So now I have a smidgeon of credibility, which is great, but what I have produced is only a very small incremental step in a different direction. How can I possibly move symbol manipulation along at a usefully large step? How can I invent something amazingly powerful? This is the question any inventor or artist would ask. And my stodgy old brain, is it still able to fluidly create new and useful connections?! I have noticed that my memory has gotten better over the last few years, which worries an artist-type such as myself. I should be able to be flexible and see things afresh. So I worry.

old man navigator

But my worry gives way to perspective. It’s not my ability to invent something which is why I must keep working and why I invite others to join me. It’s not what I build, it’s about where I am going. I am a navigator, not an inventor.

My work is to augment our ability to orient ourselves in our information, to see and make connections. These are all navigation metaphors and they apply to the process of building the tools as well.

I take responsibility of the question, of the goal, not the answer or a particular way of getting there. Sure, my world is that of visual symbol manipulation (text and associated visual communications) but to navigate we must first choose which world to venture forth into.


My future work will be to further improve the navigation and display of text in Author and on the web, including through compressed scrolling, dynamic views and more atomic authoring, including though the user of hyperGlossaries. Much of the work will be to implement small, incremental changes to make the information flow more liquid. The reach of my research and implementation–the twins of progress–will venture as far as my mind will stretch and collages will entertain.

In the distance I see a more liquid information environment where users can become deeply literate because their tools are powerful and they can view and interact with their knowledge in visceral ways, like moulding magical clay.

This is not a solo effort. It is an effort in building dialogue in addition to systems and approaches. This is what I have been doing for almost a decade now with The Future of Text Symposium and look forward to continue.

My premise is simple: Continue to work to employ the occipital lobe to support the prefrontal cortex–to use our eyes to think. And that is my journey.


In the Information War : Arm the Citizen (part 3) Pointing Out Solutions

1 | 2 | 3

For this final chapter on arming the citizenry I am writing en-route to the cradle of alphabetic writing: Egypt. After three days of being bed-ridden and sweating rivers, I am seated at the back of a Boeing 77-300 on Egyptair with good food, plenty of water (though no ice for my brought on-board UK/Norwegian iced mocha), wearing headphones over my Stetson brand baseball cap and my large hoodie over that. I’m awaiting the completion of Liquid | Author 3.5 with the ability to post to WordPress and I’m looking forward to Chris’ work on his ‘weblaux’, or liquid space, or link view or card space or whatever name it ends up being…

That’s my context. This is my point:


To truly arm the citizenry one thing which deserves serious consideration is how we can truly unleash the power of interactive text. The passive medium writing evolved over the first five millennia or so–clay, papyrus & paper–provided scope for powerful innovation which can now be further explored and developed in the truly interactive media of powerful networked computer systems.

We are still stuck within the legacy of the passive medium though, looking, as Marshall McLuhan said, at the future through the rear view mirror. In as much as the inherent interactive character of computer systems can bring real life experiences to hyper-real life, such as visually rich multiplayer combat simulations and fantastical but real looking movies, the interactivity potential of digital text offers rich opportunities.

To be clear, by interactive text I not only refer to the characters of the alphabet but also their connective potential as well as their layouts and any symbols or other visual means to connect and separate them.

Augmenting Text Interaction Augments Us

The core of what I am putting forward here I feel I can best express in bullet points:

  • Text is a visual augmentation of our long term memory
  • Basic interaction with text, such as using a pencil on paper to write and connect makes the text also more of an augmentation of our working memory
  • Making the text more interactive allows the text to augment both our long term and working memory and thus our thinking
  • In physiological terms the interactive text allows the occipital (visual) lobe to augment our prefrontal (higher thinking) cortex

State of The Art

We have seen the very early implementations of this with features such as spell check and the ability to change the look of the text through changing fonts and line breaks on the fly. We have furthermore experienced weblinks where an author can type in an address in a text document to another document and have a high degree of confidence that the reader will be able to ‘click’ in the text and that a ‘web-browser’ will proceed to connect to the location and load the document hosted there.

These are truly wondrous innovations but they are only the very surface of what we can expect to mine the potential of digital text for. Text, as Yuval Noah Harari points out, was invented to solve different problems than speech.

Digital Is Different

On paper some of the inherent qualities of text, such as permanence and linearising of thought, can also be limitations. In digital media these limitations not only disappear they open up new avenues for communicating and thinking with text.

For many who are not involved with thinking about interactive text on a daily basis much of this can understandably be a bit abstract. This is why I have followed the advice of my mentor Ed Leahy: Don’t tell ‘em, show ‘em. So here are some examples of what I am working on. Please refer to my annual symposium on The Future of Text for more work.


At an immediate and visual level we can allow the user to ‘pinch in’ when reading a document to automatically collapse the document to hide all body text and show only headings. Instead of headings and the table of contents being separate, they are now only different views of the same text. This is what we have built into our Liquid | Author word processor for macOS. Further ‘magical’, or digital-only innovations we have built include the ability to select text and hit cmd-f to hide all the sentences in the document which does not have the selected text, a very handy way to check on something which you know was introduced earlier in the document but you forgot where.

With the companion application Liquid | Flow the user can also select any text and have it looked up in any of a myriad of sources within half a second. (Speed matters: As Marshall McLuhan pointed out: The difference between still pictures and movies is the frequency of which they are updated). This is what Doug Engelbart considered to be ‘implicit’ links, in the way that a word is implicitly connected to its entry in the dictionary for example.

In collaboration with Christopher Gutteridge I am working on another one of Doug Engelbart’s ‘obvious’ functions which we are missing today, which highlights our bumpy road into the future: High-resolution links. In most cases today we can only link to whole documents. We don’t even have the ability to point to pages (particularly in commercial eBooks). This severely retards our ability to usefully assess citations which means that academia continues to work on the principles that work known to an advisor or editor is useful but work to a new source is unlikely to be consulted.

Links are one way to move around between documents (and inside documents with internal links) and so is scrolling, since page number don’t mean much in the case of digital documents. However, even scrolling is a simple legacy behaviour. We are not signalling to the system our desire to move up or down in a facsimile of a papyrus scroll when we put two fingers together on a trackpad and move them up or down. We are signalling our desire to view a different part of the document. This is where something like compressed scrolling can come in where what happens is that the body text is gradually compressed so that the user sees more headings and less of illegible body text. More opportunities are described in the blog post Compressed Scrolling including work I hope to carry out with Howard Oakley:

It is not difficult to think of how we can allow the user to ‘tear’ up the traditional, wood-block on cotton paper rectangles of text and play around with a mind map, Post It note, KJ-Ho or any other visual notes-on-a-board style thinking. As we see on countless drama series on TV, the user could turn their text into a ‘murder wall’ with any media grouped in any way the user thinks is useful with (traditionally) red thread connecting pertinent parts. This is partly what the forthcoming Dynamic Views feature of Liquid | Author aims to deliver : If we go further into the digital mode of interactions, we can choose to not only work with the text directly in front of us authored at the time, but to connect to external data sources at will, such as blogs and academic document repositories and connect and layout as our curiosity takes us, only to rein it back in when it becomes to messy, for then to add another level of connections and views. This is related to the work Christopher Gutteridge and I are doing in the WAIS (Web And Internet Science) department at the University of Southampton where I am currently a PhD student.


Beyond this small menu selection there is a huge scope of opportunities if we choose not to focus on the technological implementations, but simply use what we build to further explore what the most useful interactions can be with text, and with other media. We already have high powered computers in general use and the displays on our desktop are already at high enough resolution and large enough size to cover most of our useful visual range (any larger and we’d have to more our necks more). We need to keep developing using the best understanding we have, in order to understand the principles behind interactive text.

We already know that interactive text is interactive, which is such a core aspect of what digital text is that we named it this. Everything flows from the interactions, otherwise we are simply recreating non-interactive text in an interactive medium, such as we do with PDF documents which are only images.
We furthermore know that text is visual and thus we should keep experimenting with how the text is visually portrayed.
The inherently connective aspect of text, from simple keyword searches to linguistic analysis, is something we know and take for granted in ordinary speech which can be transposed onto digital text.

We need two ask: What is it possible to do with interactive text and what is useful, and for what? How far can we enable the user to sculpt their text to their liking? How deeply can we augment their ability to create and see connections?


I asked ‘for what’ at the end of the previous section and to me it it’s simple: The purpose of my work is to foster and give opportunities for ever deeper literacies. It’s not enough to simply know how to read and write today, we must also know intimately the tools at our disposal. In ‘The Ironies of Automation’, Lisanne Bainbridge wrote that a hidden danger of automation is that it will reduce our abilities. An example cited is how we don’t remember phone numbers anymore and struggle with handwriting. I would beg to differ though. As we climb the hill of what systems can do for us, it is both our opportunity and obligation to actively learn how to use the new capabilities handed to us. With the case of phones storing our numbers a new literacy become knowing when to call and when to text, a complimentary functionality opened up by the same smart devices.

Our ‘war’ against misinformation and navigating all the issues thrown up by the term ‘fake news’ as The Road To Unfreedom’ describes, we need to consider that our mental freedoms are rooted in the constraints and augmentations we have in how we interact with our own thoughts and those of others.

Those who does not have the ability to read is at a disadvantage to those who can, when it comes to accessing the thoughts of others and one’s own in the future. Similarly, we talk of the different abilities of someone who has barely started cobbling together letters into words and someone who is a successful author or researcher. It is not outrageous then to look at different levels of literacy of how we use the digital opportunities afforded to us at this point. And beyond.

Aspirations, Questions

Will we get to a point where a document which does not feature high-resolution citations to back up assertions seems as spurious as a schoolyard taunt?

Will we get to a point where a document which only offers one way to view it is viewed as quaint as a Sumarian cuneiform tablet?

Will we get to a point where a document which does nor explicitly make it clear to wherever it is stored what it contains and how it’s connected looks as integrated as a back of an envelope drawing?

Yes, I use the term ‘document’ since it is simply a framing of a human intention. This does not imply a lack of interaction, a lack of flow, simply a frame to hold on to.

The Dream

In the end, if the populace is better informed, better connected to their neighbour and those beyond, in deeper understanding of the contexts of their lives and of our world, I firmly believe we will see a human race with kinder intentions, wiser actions and more harmonious collaboration for a better future for all. This is not a naive position, we will not become angels with more freedom, we will still err and we will still have disagreements and lack of empathy but since it’s clear that isolating ourselves from other people and from other points of view is clearly destructive, I say that integrating and connecting ourselves can be powerfully constructive.

I don’t think we can preach freedom, we must enable it and watch the human mind bloom.