Testing

Testing

The goal of this project is to test the effectiveness of interactive text in support of Reading, Authoring and, depending on the progress of the project; Thinking.

Preliminary Research for Test Design

Before I do any testing I will need to establish what to test and how. I will therefore:

 

•  Start with the perceived requirement of interactivity on my philosophy of information and interaction  

•  Look at my own work and habits 

•  Performa a literature review of how knowledge workers, particularly postgrads do their work  

•  Send questionaries to students asking what they do and how 

•  Interview students in more depth and watch them work 

•  Etc.

The Tests

I plan to carry out two types of tests: Speed Tests and Comprehension Tests.

Speed Tests

The speed tests are simple tests to see how long it takes to carry out specific tasks, which knowledge workers do frequently, including:

Reading Focused

 

•  Looking up selected text in a reference, such as wikipedia or etymology online 

•  Searching found citations in Google Scholar, for further context 

•  Searching found citations in Mendeley or similar 

•  Finding multiple occurrences of selected text in a document or finding first occurrence and returning to place 

•  Etc.

Writing Focused

Adding citation from an academic publication, book, web resource and video

Organising and re-organising notes

Etc.

Comprehension Tests

The comprehension tests are more involved and more difficult to design but will likely have a good relevance. 

Test Considerations

These tests will need to be carried out on students who have trained in the interactive system, not simply having been given them on the day.

A single, well regarded academic document will be used, even though a literature review should include more than one document but this will likely be too much to ask of the students.

The document will be from a field  of interest to the student (likely computer science) so that it will be understandable but hopefully not one they have read before. 

The Questions 

The document will be provided to the subjects in either PDF or richly interactive digital form and there will be questions provided, which will correspond to the type of questions a reader would ask during a literature review, such as:

 

•  What are the key terms? 

•  What are the main sources cited and are they good sources (relevant, correctly cited and have high academic standards) 

•  What sources used would you use to learn more about this? 

•  What quotes/citations would you use from this document to highlight the main points? 

•  Does this author seem credible? What biases may be at play?

‘Features of Similarity’ : Amos Tversky

‘Features of Similarity’ : Amos Tversky

From ‘The Undoing Project’ by Michael Lewis.

Highly relevant to the Liquid View design.

Summary: When we compare something we judge features, not a simple physical distance of ‘resemblance’ on a mental map/chart.

 

Excerpts from the section:

 

What goes on in the mind when it evaluates how much one thing is like, or not like, another? The process is so fundamental to our existence that we scarcely stop to think about it. “It’s the process that grinds away constantly and generates much of our understanding and response to the world,” says Berkeley psychologist Dacher Keltner. “First of all it’s, how do you categorise things? And that’s everything. Do I sleep with him or not? Do I eat this or not? Do I give to this person or not? Is that a boy or a girl? Is that a predator or prey? If you solve how the process works, you solve how we know things. It’s how knowledge about the world is organised. It’s like the thread that is woven thorough everything in the mind.”

The reigning theories in psychology of how people made judgments about similarity all had one thing in common: They were based on physical distance. When you compare two things, you are asking how closely they resemble each other. Two objects, two people, two ideas, two emotions: In psychological theory they existed in the mind as they would on a map, or on a grid, or in some other physical space, as points with some fixed relationship to each other. Amos wondered about that.

“What Amos worked out was that whatever is going on is not a distance,” says Gonzalez.

“In one swoop he basically dismissed all theories that made use of distance. If you have a distance concept in your theory you are automatically wrong.”

Amos had his own theory, which he called “features of similarity.”

He argued that when people compared two things, and judged their similarity, they were essentially making a list of features.

These features are simply what they notice about the objects.

They count up the noticeable features shared by two objects: The more they share, the more similar they are; the more they don’t share, the more dissimilar they are.

From Amos’s theory about the way people made judgments of similarity spilled all sorts of interesting insights.

If the mind, when it compares two things, essentially counts up the features it notices in each of them, it might also judge those things to be at once more similar and more dissimilar to each other than some other pair of things. They might have both a lot in common and a lot not in common. Love and hate, and funny and sad, and serious and silly: Suddenly they could be seen—as they feel—as having more fluid relationships to each other. They weren’t simply opposites on a fixed mental continuum; they could be thought of as similar in some of their features and different in others. Amos’s theory also offered a fresh view into what might be happening when people violated transitivity and thus made seemingly irrational choices.

When people picked coffee over tea, and tea over hot chocolate, and then turned around and picked hot chocolate over coffee-they weren’t comparing two drinks in some holistic manner. Hot drinks didn’t exist as points on some mental map at fixed distances from some ideal. They were collections of features. Those features might become more or less noticeable; their prominence in the mind depended on the context in which they were perceived. And the choice created its own context: Different features might assume greater prominence in the mind when the coffee was being compared to tea (caffeine) than when it was being compared to hot chocolate (sugar).

And what was true of drinks might also be true of people, and ideas, and emotions.

The idea was interesting:

When people make decisions, they are also making judgments about similarity, between some object in the real world and what they ideally want.

They make these judgments by, in effect, counting up the features they notice. And as the noticeability of features can be manipulated by the way they are highlighted, the sense of how similar two things are might also be manipulated. For instance, if you wanted two people to think of themselves as more similar to each other than they otherwise might, you might put them in a context that stressed the features they shared.

Two American college students in the United States might look at each other and see a total stranger; the same two college students on their junior year abroad in Togo might find that they are surprisingly similar: They’re both Americans!

By changing the context in which two things are compared, you submerge certain features and force others to the surface. “It is generally assumed that classifications are determined by similarities among the objects,” wrote Amos, before offering up an opposing view: that “the similarity of objects is modified by the manner in which they are classified. Thus, similarity has two faces: causal and derivative.

It serves as a basis for the classification of objects, but is also influenced by the adopted classification.”

A banana and an apple seem more similar than they otherwise would because we’ve agreed to call them both fruit. Things are grouped together for a reason, but, once they are grouped, their grouping causes them to seem more like each other than they otherwise would. That is, the mere act of classification reinforces stereotypes. If you want to weaken some stereotype, eliminate the classification.

Update on Author and PhD

I have had a hard time dealing with the notion that academics generally are not interested in new technologies and Author is still too buggy for release. I have decided to spend my PhD showing, or not, that interactive text is useful. I have also made a deal with the coders of Author that I will pay a small fee up front and put the most of the fee into escrow which will only be related when/if Author actually ships. So lets’ see….