Skip to content

Category: Deep Literacy

Why work on Improving our Tools for Thought?

 

I sometimes get told that the basis of my work with continuing to develop technologies to
augment our ability to understand and be understood
(a recent wording I came across courtesy of Gyuri who referred to Maria Popova) is mis-placed, mis-guided or simply not possible or useful.

 

Questioning the Foundations

 

Here is a comment from someone I respect, just from this week: “Could be that the premise, that we can develop technologies to do as you suggest, is false. Perhaps the solution is to be found elsewhere.”

This is not very useful… Anyone who feels that there should be a difference in the world are saying that their perspective, their wishes and views are worth acting on. Actually, let me take it a bit further: When I, and all others who wish to make a change, we are naturally the unreasonable ones and, with notable exceptions, our egos are fragile. We are saying we know better than the whole world’s experts and that is something gutsy to say.

I have a few comments to the doubters and naysayers on this and I’ll put them down here with the hope that I can tag and categorise this post well enough that I can refer to it in the future.

Initial Reply

My first reply is a question: If you came across a doctor working on a patient would you say maybe the doctor should be working on another patient or a researcher on another disease? What about climate change? Is it not worthwhile to work on different aspects of this issue or do we all need to work on all of them or find the one magic bullet?

The New Medium Reply

A more thoughtful response when I have the time to engage with the person suggesting that this work is without value is to ask: In what way is it possible for the tools we use not to influence our thoughts? I leave it to the deep thinker Marshall McLuhan to put it into much more eloquent language than I ever could:

“Environments are not just containers, but are processes that change the content totally.”
Marshall McLuhan Edited by Eric McLuhan & Frank Zingrone “Essential McLuhan” Routledge 1997 ISBN 0-415-16245-9 page 275. 1967

“All media are extensions of some human faculty- psychic or physical”
the wheel
is an extension of the foot
the book
is an extension of the eye
clothing, an extension of the skin,
electric circuitry,
an extension of
the
central
nervous
system”
Marshall McLuhan The Medium is The Massage,  p 31-40

“The medium, or process, of our time – electric technology is reshaping and restructuring patterns of social interdependence and every aspect of our personal life.
It is forcing us to reconsider and re-evaluate practically every thought, every action, and every institution formerly taken for granted.
Everything is changing: you, your family, your education, your neighborhood, your job, your government, your relation to “the others. And they’re changing dramatically.”
Marshall McLuhan The Medium is The Massage

Perhaps my favourite quote:

“The alphabet was one thing when applied to clay or stone, and quite another when set down on light papyrus.”
Marshall McLuhan Edited by Eric McLuhan & Frank Zingrone “Essential McLuhan” Routledge 1997 ISBN 0-415-16245-9 page 279. 1964

This presages the unique characteristics of the unique potential of digital interactions which opens up opportunities far beyond what analog substrates could offer. We need to honour this if we are to thrive in the digital knowledge world and not drown in too much information or mis-guided by propaganda and ‘fake’ news.

“Today we are beginning to notice that the new media are not just mechanical gimmicks for creating worlds of illusion, but new languages with new and unique powers of expression.”
Marshall McLuhanEdited by Eric McLuhan & Frank Zingrone “Essential McLuhan” Routledge 1997 ISBN 0-415-16245-9 page 272. 1957

Finally, he pointed out that Socrates had concerns about writing, as written down by his student Plato:

“In the Phaedrus, Plato argued that the new arrival of writing would revolutionize culture for the worst. He suggested that it would substitute reminiscence for thought and mechanical learning for the true dialect of the living quest for truth by discourse and conversation.”
Marshall McLuhan Edited by Eric McLuhan & Frank Zingrone “Essential McLuhan” Routledge 1997 ISBN 0-415-16245-9 page 285. 1954

To this I urge us to make our knowledge, particularly out text, which houses the majority of our knowledge and is far more flexible than any other medium, to become as interactive as a real conversation, to be able to deliver on text systems Socrates would approve of, what I call Socratic Authoring.

Even Winston Churchill had the insight that We shape our buildings, and afterwards, our buildings shape us.

Other Fields

I follow this with a clear statement that of course there is more to live than tech. I do not think that technology should be ignored though. Yes, we will need better management philosophies and principles, more work on empathy and more humanitarian work. Yes we could do with a unifications of the worlds Abrahamic religions into a uniform whole which preaches love to all not just those of a chosen denomination. Yes, we would be better off being vegetarian and yes wars and social and economic injustice must be tackled. There are many issues we as a species must address if we are to survive, let alone thrive.

 

Please allow me a few sections on why making more powerful tools for thought is a valuable endeavour:

 

The Importance

 

My friend and mentor Doug Engelbart wrote in his seminal 1962 paper:

I maintain that the goal of the information war should be to empower us, the people, to better get to grips with information in the world and in our heads in order to developed more connected, less myopic world views and as a consequence develop societies which deliver more on the teachings of religions and moral philosophies: More love for our neighbour, more respect for our environment. Love and respect both come from connections and we therefore need to develop technologies to help us connect, not alienate.
Engelbart, 1962

He further presented in Singapore decades later something I feel is so powerful and powerfully important I will quote him here at length:

The feature of humans that makes us most human – that most clearly differentiates us from every other life form on Earth – is not our opposable thumb, and not even our use of tools. It is our ability to create and use symbols. The ability to look at the world, turn what we see into abstractions, and to then operate on those abstractions, rather than on the physical world itself, is an utterly astounding, beautiful thing, just taken all by itself. We manifest this ability to work with symbols in wonderful, beautiful ways, through music, through art, through our buildings and through our language – but the fundamental act of symbol making and symbol using is beautiful in itself.

Consider, as a simple, but very powerful example, our invention of the negative – our ability to deal with what something is not, just as easily as we deal with what it is. There is no “not,” no negative, in nature, outside of the human mind. But we invented it, we use it daily, and divide up the world with it. It is an amazing creation, and one that is quintessentially human.

The thing that amazed me – even humbled me – about the digital computer when I first encountered it over fifty years ago – was that, in the computer, I saw that we have a tool that does not just move earth or bend steel, but we have a tool that actually can manipulate symbols and, even more importantly, portray symbols in new ways, so that we can interact with them and learn. We have a tool that radically extends our capabilities in the very area that makes us most human, and most powerful.

There is a native American myth about the coyote, a native dog of the American prairies – how the coyote incurred the wrath of the gods by bringing fire down from heaven for the use of mankind, making man more powerful than the gods ever intended. My sense is that computer science has brought us a gift of even greater power, the ability to amplify and extend our ability to manipulate symbols.

It seems to me that the established sources of power and wealth understand, in some dim way, that the new power that the computer has brought from the heavens is dangerous to the existing structure of ownership and wealth in that, like fire, it has the power to transform and to make things new.

I must say that, despite the cynicism that comes with fifty years of professional life as a computer scientist, inventor, and observer of the ways of power, I am absolutely stunned at the ferocious strength of the efforts of the American music industry, entertainment industry, and other established interests to resist the new ability that the coyote in the computer has brought from the heavens. I am even more surprised by the ability of these established interests to pass laws that promise punishment to those who would experiment and learn to use the new fire.

As the recipient of my country’s National Medal of Technology, I am committed to raising these issues and questions within my own country, but I am also canny enough to understand that, in the short term, it is the nations with emerging economies that are most likely to understand the critical importance and enormous value in learning to use this new kind of fire.

We need to become better at being humans. Learning to use symbols and knowledge in new ways, across groups, across cultures, is a powerful, valuable, and very human goal. And it is also one that is obtainable, if we only begin to open our minds to full, complete use of computers to augment our most human of capabilities.
Engelbart, 2002

 

If You Agree, Now What?

 

So, dear reader, if you are convinced that investing in the tools we think with is a worthwhile pursuit, please allow me to outline issues and proposals for how to do this:

 

The Market

Simply expecting the market to provide ever more powerful tools for thought is not realistic and if we look at the very slow development of something as basic, no, something as fundamental to daily knowledge work as a word processor, there has been incredibly little improvement since the early work Doug Engelbart demonstrated in 1968. We are still experiencing the powerful surge of Moore’s Law (which he formulated based on a discussion with Doug Engelbart, as retold by Jeff Rulifson) which makes our computational hardware immensely more powerful with every generation. My new ‘smart’ phone, the iPhone XS max is capable of 5 trillion operations a second and what’s called ‘triple a’ title computer games, such as ‘Battlefield’ not only looks near-hot realistic but also features rich interactions.

Doug further said, reflecting on the question of how the invisible hand of the market would solve the problems, in the same address:

One possible response to my examples is to say, “Doug, be patient. These are new problems and hard problems and it takes time to solve them. We will have better tools and better laws over time. Just wait.”
Engelbart, 2002

We have waited. Compared with Doug’s NLS/Augment we have spell check and nice fonts, both of which are useful but we are still lacking basic functionality developed fifty years ago and which ease-of-use and commercial constraints consciously dumbed down.

We lack high resolution addressing, rich view specifications and more. These are not simply legacy features of an age old system, these are the foundations necessary to build our future knowledge worlds on.

He continued:

Oxymoron: “Market Intelligence” – One of the strongly held beliefs within the United States is that the best way to choose between competing technologies and options for investment is to “let the market decide.” In my country we share a mystical, almost religious kind of faith in the efficacy of this approach, growing from Adam Smith’s idea of an “invisible hand” controlling markets and turning selfish interest into general good. The “market” assumes the dimensions of faceless, impersonal deity, punishing economically inefficient solutions and rewarding the economically fit. We believe in the wisdom of the market and belief that it represents a collective intelligence that surpasses the understanding of us poor mortal players in the market’s great plan.

One of the nice things about getting outside the U.S. – giving a talk here, in Singapore, for example – is that it is a little easier to see what an odd belief this is. It is one of the strange quirks of the U.S. culture.

In any case, it is quite clear that whatever it is that the market “knows,” its knowledge is fundamentally conservative in that it only values what is available today. Markets are, in particular, notoriously poor judges of value for things that are not currently being bought and sold. In other words, markets do a bad job at assessing the value of innovation when that innovation is so new that it will actually rearrange the structure of the markets.
Engelbart, 2002

The market serves what can be researched and sold. The market has no mechanism for providing what might be the best investment for the consumer long term.

Academia

Academia certainly has something to contribute to the field but academia is built on standing on shoulders, not giant leaps. The very structure, the fabric of academia is to always cite prior work and to increment only in small steps. This is important and useful but it has not been able to produce new insights for powerful tools and systems and to take them into general use.

 

What We Need To Do

Once we accept the importance and the issues we need to take ownership for how to deal with this and to provide the tools to enable deeper literacies to emerge: http://www.deep-literacy.com

That is outside of the scope of this article but has been extensively discussed on this blog as well as elsewhere. I would be happy to engage in thoughtful dialogue on the subject. You can contact me at frode at liquid dot info.

1 Comment

An end of summer update

I have now settled into my new home with my beautiful wife and wondrous baby boy, who is now in nursery 3 days a week and the shock of loosing my father has turned into a deep sorrow I understand from friends who have been through it I will simply have to learn to live with, and I accept that to a degree. This all means that I have 3 days a week of relative calm to work, in my new home office without domestic or other obligations pulling at me.

Liquid | Author

I am continuing to develop my Mac word processor Liquid | Author but now with a more commercial angle since I can no longer expect the odd, random investment from my father, now that his estate is being broken up and starting to come out of probate I feel more like the money he left for me, and thus my family, is like holding a bag of sugar; I am very concern that it should run out, little by little, and my focus in life now is of course to provide for Edgar. Author is receiving some work to make sure that citations are handled as cleanly and clearly as possible and then work on Author will pause so that I can market it and make it at least some kind of level of sales success. The only other thing we are adding to Author is the ability to post to WordPress and this is because of the jrnl project:

jrnl

I am co-hosting the 50th Anniversary of Doug Engelbart’s demo with ‘The Engelbart Symposium’ on the 9th of December in Silicon Valley (Wendy, I think we’ll be asking you to be on a panel, it’s almost all panels, no real presentations, but we have not finalised the program yet). The other hosts are The Engelbart Institute under Christina Engelbart, Vint Cerf of course and The Computer History Museum who are also our hosts. I will be hosting a panel discussion that day and we will have an area set up for demos, of which my work will be one station. We will also produce a book of the day, which will contain all the contributor’s transcribed presentations/panel discussions and a short pre-or-post written statement by them, along with monochromatic pictures. It’ll be nice. We will also be posting this contents online in a way which honours and demonstrates some of Doug’s ideas, primarily high resolution addressing and viewspecs:

For this I the started the jrnl project (pronounced Journal, I just could not get a good domain name with all the letters showing up for duty) which recalls Doug’s Journal and is based on WordPress for rapid opportunities for actually making something useful. There was an earlier project started when I hosted http://doug-50.info/index.html which I hoped could produce something wonderful from a diverse group of people but it became a 100% knowledge graph effort due to the passion of the strongest contributors so whereas I am still involved in some of their dialog and they will exhibit on the 9th, this is not something I feel strongly connected with or able to understand. There is one important overlap between their work and the jrnl project and that is hyperGlossaries http://thefutureoftext.org/hyperglossary.html

I am working with Chris Gutteridge of Southampton on the jrnl project and Gyuri Layos who is also actively involved with the knowledge graph work, on the hyperGlossaries and Chris is also working with me on the jrnl project along my coders in the Ukraine who built the current version of Liquid | Flow, my Mac text-interaction application. I am also hiring external wordpress expertise to make sure we are going about the plugin system in the best way.

I further had a meeting with professional, full-time wordpress developer Shane Gibson who is considering donating some time to the project since I told him I have no money for this but he is excited to be working on a project with Vint’s name attached, which shows commitment to the ideals here I think.

The interaction method is a blue dot which appears when user selects text. Point to it to get a menu of options. This is what used to be called the Hyperwords Project and was developed for Mozilla and Chrome as well as for wordpress which is the code we will be using now: https://jrnl.global/2018/09/21/refined-blue-dot/

What I feel would be the minimum features inspired by Doug to demonstrate (as in show fully working and let people download and install as robust plugins for their own wordpress setup) are high resolution addressability (in the form of the ability to Copy As Citation which provides at least a paragraph level address in the form of an anchor so that anyone can use the link in any context and it will work) and some ViewSpecs, likely an Author style Find In Page (which re-draws the page to show only sentences with the selected text) and/or Flow (which redraws the text with line breaks after , and double line breaks after . as discussed in: https://jrnl.global/2018/09/09/visual-syntactic-text-formatting/

PhD

In an advisor meeting with Les when I was still fully active we agreed that my interest in what Doug Engelbart called ViewSpecs should be the focus of my research but with a focus on the act of changing views, not just on the views themselves.

The act of changing a view should certainly not be a removed act like clicking a button but rather an action where the user is manipulating the shape of the information. I came up with the notion of Compressed Scrolling this week. The idea is that when the user starts to scroll past a certain speed (going from positional scrolling to navigational scrolling in Chris Gutterdidge’s words) then currently all the user sees is a grey block of illegible text. What I propose is that on this threshold the view changes to hide generic text and highlight useful text since the user is now in seeking mode. The user does not start to scroll because they want to move the document up and down, they start to scroll to see another part of the document.

So what could happen is that:

  • All the body text becomes greyed out and reduced in size making headings more prominent
  • All names in the document stay black and the rest of the text greys out
  • All the names in the document become icons if they are for companies and pictures if for people
  • Doug Engelbart suggested colour coding text based on the category words. For example we did a fun test together where all the words about tech were coloured yellow, companies blue, people green and so on. It really did give an insight as to what sections discussed what since it didn’t really take that long to learn the colours but it was ugly and not so readable when you stopped to read. In this scenario however, the colours would only be applied when scrolling, not when reading

We are further working on realising a powerful hyperGlossary in this environment: http://thefutureoftext.org/hyperglossary.html

These are potentially powerful views to help the user move around their own or other’s documents. I have been in touch with Howard Oakley who blogged about macOS Mojave’s text analysis APIs and we’ll try to look at some of this for Author, while Shane might look into the for jrnl wordpress and Chris is interested in both. I met Howard through Twitter, having Mark Anderson as a mutual friend. This is the post I read which opened my eyes and got in touch with him: https://eclecticlight.co/2018/10/01/getting-word-frequencies-by-lexical-class-nalaprop-gets-more-useful/

I think this line of enquiry could be useful and a good PhD, but it would real user testing so I will need to get some research funding, which I am working on.

 

1 Comment

Voice Interaction for Text

I have long argued against voice interfaces for information manipulation since they interfere with the visual-dexterity operations of reading and writing. There is good reason why no-one has asked for a system where you speak ‘turn the page’ to turn the page since it would take you out of your internal mental world and break your flow.

However, there are interesting developments being opened up, such as the linguistic support for macOS Mojave which Howard Oakley mentions in his blog https://eclecticlight.co/2018/09/27/mojaves-linguistic-support-a-promising-start and Apple’s APIs for machine learning being included in iOS and macOS, as coreML https://developer.apple.com/machine-learning/ provide opportunities for very rich text manipulation.

What can be reasonable to design systems for now includes such interactions as:

  • Show me only sentences with names,–remove headings,–no, not these names over here, ignore them–highlight ‘Steve’ in red.

Designing the interfaces for this can quickly include a lot of buttons or commands to memorise and that is an issue.

However, I recently splurged and bought an Apple iPhone X S max which is very, very fast (capable of 5 trillion operations per second!) and which processes speech commands near perfectly and near instantly.

It is becoming clear that we must start experimenting with flexible views based not just on linear commands but also on analysis (coreML) and interactions for this can benefit from speech, where the system does not need a ‘hey Siri’ prompt and which is aware of the on-screen/in-document data it is working on, including the results of previous operations for continual builds of views.

This could be backed up with a tokenised command bar where the commands are added as tokens when spoken and acted upon, both to make sure the user is happy the commands are correctly interpreted and also for the user to then visually edit them as desired and share/save the command set as a ViewSpec if useful.

I feel this warrants serious consideration.

 

1 Comment