Tollefsen, D.P. (2009) WIKIPEDIA and the Epistemology of Testimony. Episteme 6 (1), 8-24.
Wray, K.B. (2009) The Epistemic Cultures of Science and WIKIPEDIA: A Comparison. Episteme 6 (1), 38-51.
Sanger, L.M. (2009) The Fate of Expertise after WIKIPEDIA. Episteme 6 (1), 52-73.
Magnus, P.D. (2009) On Trusting WIKIPEDIA. Episteme 6 (1), 74-90.

In this set of articles, philosophers considered the implications of Wikipedia on theories of testimony, taking various positions on its role and significance as a knowledge source. A common theme is what our use of a collaborative but mostly anonymous system tells us about human credulity. A further thread is how many of the cues we normally use to assess credibility are not available in Wikipedia. Although there still ways to determine how accurate its articles are, we don’t actually tend to use them in practice. In taking the philosophical tack, the authors join up our current knowledge seeking habits with traditions within philosophy, but also remind us of the huge power and reach of Wikipedia.

In the first article, Tollefsen claims that Wikipedia can count as testimony on the assurance principle – that the authors are providing some assurance about the correctness of the content. Moreover, it has many qualities of group testimony as distinct from the aggregation of individual testimonies. She notes that the more well established articles represent a balanced view of the topic (as we’ll see this is perhaps overidealistic and is not entirely corroborated by the other authors.), seen in a way as the settling of the group mind on an issue. As we don’t know the individuals or groups involved in authorship, we are also to some extent placing our trust in a system.

Tollefsen largely supports an ‘antireductionist’ approach to justification, where you you are justified in believing others as long as you have no reason to believe them insincere. In the case of Wikipedia, however, she concludes that default entitlement to trust does not hold, and that we need to treat Wikipedia more cautiously, like “talking to a child”. Later on, she notes that we monitor testimony by checking it against our background beliefs, and this is how we may choose to accept the content of articles.

Wray takes the approach of comparing the process behind wiki articles to the practices of scientists, and comes out more in favour of science as a sound knowledge making practice. He observes how hard it is to authenticate online claims, quoting Shapin that “trust is ubiquitous in knowledge societies”. He further notes that the reliability of wikipedia is based on ‘invisible hand justification’ – in that there is no one person in charge of quality assurance on an article.

Wray sees wikipedians as having less to lose than collaborating scientists and therefore less incentive to be honest and conscientious. That said, he does go on to note that the rush to publish in science is a knowledge anti-pattern and may lead to as many accuracy issues as the rush to post online, which he casts as a form of gossip.

Wray compares egoism and non-egoism as justification in testimony. An egoist approach would say that you need to know the character of the person testifying, and a non-egoist would say it is just enoughto be told someone else believes something. But he concludes that neither extreme really holds in Wikipedia – we dont know the reputation of the authors nor if they really believe what they tell us.

Sanger, a philosopher as well as one of the originators of Wikipedia discusses whether experts still needed now that non-experts can so effectively aggregate knowledge. He concludes, however, that the role of the expert has been too far downplayed in Wikipedia, as experts may be pushed out or ignored.

In conflict with Tollefsen’s views about the “balanced” article emerging over time, Sanger claims that article quality may be a function of the persistance and aggression of the article’s followers, not just the lifetime of the article. In content wars, experts may be driven off by aggressive amateurs with more time on their hands.

Magnus asks what it means to trust Wikipedia as a knowledge source. He notes that in the famous (though controversial) Nature study, Wikipedia proved more variable and slightly less accurate than Brittanica, though perhaps in Wikipedia omissions are more common than inaccuracy.

Magnus provides a useful list of cues that we may use in assessing written testimony including: authority, plausibility of style and content, calibration and sampling with usual sources. He notes that these come undone on Wikipedia. Authority is hard to determine – and not guaranteed with referencing. Style can be edited out. There is a real risk of plausible falsehoods in the content. Calibration may only occur on widely known, as opposed to more specialist portions.

Magnus offers some solutions: Follow sources, interpret the palimpsest – the evolution of the article – from the discuss pages, and link to the dated version of the page. All things that are possible, but even academics probably don’t do.

While the more cautious views on how we “should” treat Wikipedia presented in this journal are coming from experienced academics, I am fairly sure that the position of the average user – including students and teachers – is much more toward anti-egoism and anti-reductionism. Thomas Reid’s principle of credulity – that we are predisposed to trust our fellow man – certainly holds in the large part, especially as often there is not much at stake if we are taking on false beliefs. As the need for correctness increases, however, then so does the need to spend time (and cognitive resources) in assessing credibility.

One area that I felt was not well covered in these papers was the tendency for online information to be recycled. The verifiability principle mentioned by Sanger works well, though often means that large sections are lifted directly from pre-existing sources on the topic, many of which are available online, many of which are written by “experts”. I expect that there is research available on the extent to which Wikipedia articles are largely paraphrased prior texts, and will report it here if I come across it.

“The coffee offer forced me to focus on what I would really like to happen, instead of either doing nothing or passively accepting what someone else would like to happen”

9 Jun 2010, UCL

A set of interesting talks from researchers, museum staff and a technology supplier on digital heritage. Common themes were enhancing interaction and engagement with collections through digital media and the crowdsourcing of transcription work with its attendant risks and benefits. Crowdsourcing is seen having particular potential at a time when it is hard to secure funding for digitisation and online projects.

1. David Arnold, University of Brighton

David Arnold presented on advances in 3D object representations for heritage collections. Not really my area this, so I only picked up a few points:

• 3D Coform Project – part of big EU project with museums.
• Phased adoption of technology – first phase just reproduces what you do manually. Subsequent phases only possible with tech.
• Every time we digitise, we create potential legacy problems with the data

Tools:

• triangulated laser scanning, KULeuvan ARC 3D tool
• Sourceforce FOSS mesh processing

Example: publicscupturesofsussex.co.uk – map + 3d pics

Metadata: 3D COFORM adopted CIDOC-CRM & extension to “Digital Provenance”

2. Andy Hudson-Smith, UCL (cf Digitalurban blog)

Andy spoke about a new project aimed to attach memories to objects: Tales of Things and electronic Memory (TOTeM). Site is: talesofthings.com

You take a photo of an object/building etc.. of personal significance to you, upload it to their website and get a QR code sticker. API under development.

The object is then tagged and you can add your memories to it via the web. Object also tweets on updates. Audioboo for the audio tagging.

Apps: talking Oxfam objects (what happened to my old trousers?), artist adds QR to art to record people’s reactions to his art.

Plan to make QR code out of mosaic tiles for Shoreditch church.

“Tagging is Connecting” paper M/C journal

Follow project at twitter.com/talesofthings

Alzheimers link – triggering memories in sufferers.

3. Melissa Terras, UCL

Melissa spoke about her “Transcribe Bentham” Project & Crowdsourcing the transcriptions.

Jeremy Bentham – large amount of papers & material on him at UCL.

Crowdsourcing roots. Often those outside institutions took lead in exploiting new tech. Old idea – e.g. metal detectors revolutionising archeology with uptake by enthusiasts.

Holley, R D-Lib Magazine. “Crowdsourcing: how and why should libraries do it?”

Crowdsourcing successes:

• Galaxy Zoo – 60 million galaxies classified
• Australian Newspapers Digitisation Program www.nla.gov.au/ndp. 9000+ members thousands of lines corrected
• V & A – crowdsourcing best crops of images from collections. 14,000 to date.
• Family search indexing of family trees www.familysearch.org

and many more..

Learning from crowdsourcing experience:

• Majority of work is done by 10% of users
• Personal interest & reward – use of cognitive surplus
• Pensioners, disabled, terminally ill particularly keen
• Builds up IT expertise, “addictive”, rewards = ranking

.

Rose Holley’s checklist (on dlib). Launching system in July.

www.ucl.ac.uk/transcribe-bentham

..system will be based on mediawiki

Issues :

• linking transcription, catalogue & image
• Undoing vandalism
• Quality control

#TranscriBentham

4. Fiona Romeo, National Maritime Museum

Fiona also presented a range of NMM digital projects including crowdsourding of transcripts also.

“Your Paintings” project with BBC. Public Catalogue Foundation.

NMM photos on Flickr. Get foreign language tags as well as semantics (content related) . Some additional annotations. Get questions, which helps guide information design. Indicommons project, make it easier to share content.

Naval-history.net Naval history community – connected with them and got them to work on a set of scans – CC sharealike licenced.

NMM have a new wing under development dedicated to digital interaction and a number of interesting ideas for social sharing and enhancement of objects in the collection.

5. Sascha Curzon, Gallery Systems

Sascha presented the eMuseum federated search system, a nonprofit project which enables museums and galleries to expose their catalogues, enabling end users to search across collections. The aggregated data will also be available as XML and RDF.

I only discovered this book recently but it contains some interesting insights into group decision making, providing a range of evidence to support — often counterintuitive — observations about the best conditions for knowledge creation. As such, it relates well to Alvin Goldman’s “veritistic selection practices” — social design aimed at maximising the truth value of aggregated opinion.

Produced at a time when prediction markets, blogs and wikis were attracting growing interest, it is the book’s analysis of the more traditional social collaborative processes that stood out for me.

A central idea early in the book is the Condorcet Jury Theorem, which states that — on binary and numerical judgements at least — a group is more likely to be correct than an individual under the conditions that:-

1. Majority rule is used
2. Each individual is more likely than not to be correct

Notably, only some of the group need to know the correct answer, as the rest will answer randomly preserving the effect. This is redolent of Millionaire’s “Ask the Audience” option, which another book The Wisdom of Crowds claimed is correct 91% of the time, more often than the phone a friend option. Some warn against calling this mathematical effect “wisdom”, however.

Sunstein goes on to provide a range of evidence supporting the observation that deliberation does not lead necessarily lead to more accurate outcomes than polled, statistical samples. This may be due to:-

1. “hidden profiles“, or accurate information available to a group which is not accessed in deliberation (perhaps because only a few members of the group possess it, or are reluctant to go against the grain and challenge the majority view);
2. The “common knowledge effect“, where shared knowledge has more of an influence on the group’s decision than information held individually;
3. Informational and reputational cascades, where individuals change their view or the willingness to speak out after hearing the views of others;
4. Group polarisation, where members of a group end up with more extreme views — in line with their initial tendencies — after debate.

What this all mean for knowledge creation on the social web? Perhaps that giving scores and weighting to reputation, while establishing authority and expertise, may also dissuade or detract from informed dissent, and that such dissent may also warrant recognition.

In another review of the book, Ethan Zuckerman noted:-

“A large number of the most interesting projects taking place on the internet use strategies to aggregate information from multiple users to create new knowledge …. analyzing these systems in terms of their effectiveness in getting people to reveal hidden knowledge is, in my opinion, an excellent framework for evaluation.”

If theory and research are meant to inform practice, recommendations need to be current, relevant, and actionable. This can be hard to achieve in a domain that develops so quickly. Checkland (1998) observed a while ago how it is very hard for theory to keep up and be informative, thereby failing to match the “ideal” dynamic:

Checkland's ideal for theory informing practice

Whilst a range of research-theoretical stances can be taken, justified and accepted by the research community, only some seem to be close enough to practice to usefully inform it. This is where the researchers are either experimenting directly through design, or are involved enough to really engage with the language, problems and possible solutions in the domain. So I would favour interpretive or action research approaches in this regard:

Levels of engagement and theoretical stance (based on Aakhus & Jackson (2005))

Whilst the more descriptive, long view can still be valuable, it is hard to realise this value unless the knowledge gained is acted on by those implementing new designs and approaches. As they are the ones reading the research, this should perhaps be the research community itself in the first instance. Design improvement can then be further communicated through demonstrators, but these need to be compelling enough to be taken seriously.

References

1. Aakhus, M. and Jackson, S., 2005. Technology, Interaction and Design. In: Fitch, K.L. and Sanders, R.E.( eds) Handbook of Language and Social Interaction. LEA, pp. 411-433
2. Checkland, P .and Holwell ,S. 1998. Information, Systems and Information Systems: Making Sense of the Field John Wiley & Sons, Inc. New York, NY, USA

Originally reviewed for the BCS Information Retrieval Specialists Group Newsletter “The Informer“

This volume contains the proceedings of the 2nd International Conference on Metadata and Semantics Research, held in Corfu in October 2007. The conference had a 50% acceptance rate, with most of the presented papers being published in the book.

Despite the fact that many papers present only interim or early research work, the volume provides a fascinating overview of a cross section of metadata and semantics research in both pure form and as applied in a variety of domains, including those of cultural heritage, education and agriculture. In addition to the advances in these individual fields and domains, one is able to determine some general trends and patterns running through the work presented. Perhaps overriding all is the sense that many researchers and archivists are struggling with the problem of balancing the complexity of describing information resources within a domain adequately, while at the same time ensuring that discrete sets of knowledge can actually be successfully linked and that systems which can guide the user in retrieval and discovery can be usefully developed on top of these structures.

Immediately apparent as one reads the volume is the dominance of semantic web approaches to metadata and knowledge modelling, influencing migrations from stand-alone formats and conventions to those based on semantic web standards such as RDF and OWL. Sections are devoted to semantic web applications and to ontology engineering. In several places one sees how such approaches can provide the missing semantic aspect to rather more well-established syntactic rules and begin to enable reasoning and knowledge discovery through the use of ontologies. In Semantic Application Profiles: A Means to Enhance Knowledge Discovery in Domain Metadata Models, for instance, Koutsomitropoulos and colleagues show how the CIDOC Conceptual Reference Model for cultural heritage can be represented in OWL and used to discover related works (using a painting example). In Capturing MPEG-7 Semantics, Dasiopoulou et al, argue that a semantic model is needed to enhance and disambiguate multimedia mark-up enabled by the MPEG-7 format. Also very much worthy of mention in this regard is good the article by Voss, Encoding changing country codes for the Semantic Web with ISO 3166 and SKOS which demonstrates with a grounded example how changes and versioning can be accommodated using the RDF based Simple Knowledge Organisation System. This is clearly paramount when considering the provision of globally applicable yet cohesive and adaptive data sets.

The complexity of and unwieldiness of ontologies has been an issue in the past and several of the papers address this by providing either automated means of deriving ontology structures or reviewing tools used (or needed) for ontology and semantic web data manipulation.

In the first category, Ensemble Learning of Economic Taxonomy Relations from Modern Greek Corpora by Kermanidis shows how a number of clustering algorithms can be applied in combination to arrive at quite a high level of accuracy in deriving relations, based only on the learning sets themselves. In A new Formal Concept Analysis-based learning approach to Ontology Building, Jia and colleagues promote the FCA approach in providing a lattice of concepts rather than the hierarchy given by other popular clustering methods, which they argue as being more “true to life”. Jia et al then show how the derived ontology can then be used to provide a query refinement/expansion method.

In terms of user tools, Cardoso provides a useful state of the art in use of software to manipulate RDF data and ontologies in The Semantic Web : A mythical story or solid reality , and Enokksson et al address the problem of the remote and in-place modification of RDF in An RDF Modification Protocol based on the needs of Editing Tools.

While there are a number of papers that demonstrate derivations and developments of specialised or extended ontologies and metadata schemes, I found the more pragmatic papers to be more valuable and progressive for the way they manage to remain in touch with the ultimate aim of their work. A nice balance between the convenience of machine cataloguing and the need for oversight of human cataloguers is struck in Whitelaw and Collins’ Pragmatic support for taxonomy-based annotation of structured digital documents. The authors describe the derivation of metadata for Open University course documents (based on IEEE Learning Object Metadata) using a series of more- or less automated approaches and then having these evaluated by library cataloguing staff. The optimal solution was found to be where terms are suggested from the vocabulary based on document content, but where the staff member can click through to the definition before confirming the use of the term as metadata. The authors also begin to look at the cost benefit ratio of collecting more detailed metadata to a slight increase in cataloguing effort. While these authors don’t get as far as exploring the efficiency of retrieval of documents, this is handled in Cervera et als’ Quality Metrics in Learning Objects which stresses the reusability objective and notes that quality metrics are crucial to the navigation of resources. Notably, Cervera et al describe the need for socially based rankings and ratings to be incorporated into the metadata scheme.

A further example of attempts to enhance the usability and comparability of metadata is provided by Nilsson et al in Formalizing Dublin Core Application Profiles: Description Set Profiles and Graph Constraints. A number of practical measures for introducing conventions for how metadata elements are used and coded are presented, including standard templates and the use of a wiki to provide for easy human editing of profiles which can then also be exported at XML, thereby bridging the gap between qualitative and more structured usage guides.

Against this general background of attempts to standardise, enable interoperability between collections and provide enhanced semantic infrastructures you have the inherent messiness of human decisions as to meaning and the complexity of the encoding act itself, something that is probably not discussed enough in this community but which is covered in Scifleet et al’s The Human Art of Encoding: Markup as Documentary Practice. The authors note that metadata cataloguing is socially situated and may often be idiosyncratic, though the extent to which this may affect quality and comparability is poorly known. They report on a programme of research to study this variety in the practice of participants in the Text Encoding Initiative (TEI). Unfortunately their full findings are not covered in this volume but are well worth following up for those interested in this important question.

In summary, this is an interesting and varied read, with enough “current state of the art” review papers to interest the non-specialist as well as some good fodder for those already wrestling with metadata, semantics and knowledge representation issues within particular domains.

To see them in the context of the conference programme, I converted the tweets and the event schedule to XML suitable for display in a Simile Timeline. Click the screenshot below to access the resulting timeline mashup (nb tweets take a few seconds to load).

What would have made this exercise easier, and the conference tweets easier to track:

• If the conference programme had been published in a more mashable format eg iCal / XML
• If there was a way to identify a particular conference track (a sub hashtag? – or is that overkill?)
• A way to distinguish critical comments, reportage, and general flim-flam. Though perhaps that is part of the fun!

The graphic below shows the Bristolian IDs with the most friends who are also located in Bristol. The most highly connected accounts are in red towards the centre. The highest number of Bristolian friends was 183 (bristolnews), with the average number 11, though stats like this of course ignore the fact that many people don’t report their location (or do report it as lat/long). An initial look shows that “traditional” media corporate – and individual – IDs are well linked in, but that social and creative orgs. and those backed by online community sites are also among the most popular.

When I get time, the plan is to categorise IDs by sector and by individual / corporate (though there is a large number of in-betweens). There are also signs that the accounts are organised into four distinct clusters and it will be interesting to see if there is any discernible basis to this.

Here is a Wordle of the user bios. Not too many surprises here – a good overview of Bristol’s online creative community – though the prominence of tea was interesting (23 mentions) in comparison to coffee (8) !

In this chapter of the Annual Review, Fallis argues for social epistemology as the best philosophical underpinning to information science. Providing access to society’s knowledge through recorded information is argued as the main reason for information services. Interestingly, social epistemology was first coined in information science (by Shera) , though was also developed as an idea independently within philosophy.

Fallis covers the debates within philosophy as to what constitutes knowledge, and how these have been viewed by information scientists, most of whom have naturally taken more pragmatic perspectives. He also notes the different (classic v revolutionary) views of social epistemology, with the former a social extension of the established idea of knowledge as justified true belief, but the latter taking the social constructivist line. A further useful distinction made is between social epistemology as a normative project — the goal of which is to maximise knowledge acquisition (or at least maximise access to authoritative information) — and the sociology of knowledge, which is about understanding and describing how social factors contribute to knowledge making.

In outlining debates about necessity of the truth condition, Fallis notes that librarians will help the user try to discover the “truth”, which the user in turn sees as the key objective of using the service. That said, it is acknowledged that the collection owner cannot possibly know whether all the facts contained therein are true.

Some interesting and important conclusions come from acknowledging the centrality of social epistemology in this way:

1. That epistemic principles should guide collection organisation (e.g. by type of claim and evidence offered);
2. That information providers should collocate a range of viewpoints on a topic to allow the seeker to better determine the truth (a principle taken from Mill’s On Liberty);
3. That any aspect of an information service that enables the social acquisition of knowledge is as important as any other (sofas and coffee bars are as important as books)- I liked this one!

I felt this to be a very rich and balanced article with many considerations of lasting relevance as we move into the digital age. I also really liked the exploration of the underlying goals of information services, a topic that needs to be foregrounded more often.

Bill Gates’ keynote began by focusing on the big problems in development that his Foundation is targeting — childhood mortality, immunisation and education. Perhaps to deliberately deemphasise ICT, he introduced it only later in his talk, where he talked about scalability and learning to recognise those projects with a genuine impact while recognising failures (citing Digital Green and M-Pesa as successful examples). In doing so, there was some feeling that he was not giving due regard to the importance of localised, context-dependent solutions (eg Richard Heeks’ response)

Ricardo Legorreta's impressive Carnegie Mellon building in Doha

Earlier in the day, a very interesting study by Melissa Ho and colleagues on ICT for health in Ghana had compared a top-down, government-sponsored solution with a more bottom-up practitioner-led initiative. Both solutions dealt with communication and referrals between health care professionals, the former being based on a web platform and the latter a mobile solution. The study found more uptake with the latter, though interestingly this also seemed to have some unintended consequences, such as doctors’ phones going off at inopportune moments. This seemed to typify the delicate balance between over-structured “official” systems which struggle for uptake and the unstructured, occasionally chaotic outcomes with a user-led approach. The interesting question for me is whether such community-led work can develop sufficient structure and conformity for the solution to work efficiently into the long term.

Among the other (mostly very good) presentations that stood out for me was that by Aishwarya Ratan et al from Microsoft Research on Kelsa+, which did an ethnographic study of off-duty workers using a PC, capturing some compelling footage of workers teaching themselves basic computing through peer observation and experimentation.

The poster presentations and system demos were great, and although I didn’t get round to all of them I had a very good live demo of Frontline SMS linked to Ushahidi for participatory crisis reporting and a look at the Moca system for remote diagnosis in healthcare.

Carlos Braga giving his closing keynote

In his closing keynote, Carlos Braga of the World Bank talked about the global financial crisis and how it will continue to hit developing countries hard. While acknowledging the role of ICT in the credit crisis (through enabling complex but opaque derivative calculations), it is increasingly shown to play a strong role in economic growth. Braga compared the development trajectories of South Korea and Ghana, attributing a large amount of the divergence between the two on the better harnessing of knowledge in Korea. Braga also made use of the excellent animated bubble chart for GDP growth from Gapminder World

In sum, it was a privilege to get to sample the range of quality research and innovation going on in the ICT4D space. While the theoretical navel-gazing was occasionally wearing, there were also real signs of an emerging, more balanced synergy between the power of ICT and the headline issues in global development.