Some Stack Overflow questions

The Mamykina paper uses a nice mixed methods approach to look at both the interaction patterns on the site and the motivation of users and community owners. Some key design aspects highlighted are that the community managers are themselves respected domain experts, and that they have taken an evolutionary, collaborative approach to the development of the platform. In terms of site activity, the authors note that a high proportion of questions receive at least one answer (92%) and that answers are received and accepted quickly (median 11 minutes to be received, 21 minutes for an answer to be accepted). Whereas some users only ask (23%) and others only answer(20%), a good number do both (21%). They go on to recognise distinct user profiles in the community: activists, shooting stars, low-profile users and lurkers/visitors. Activists and shooting stars – active for a short period – together provide a large proportion of the answers despite representing a low proportion of the community (figure).

Answer behaviours (data from manykina et al)

Mamykina et al also note how the gaming aspects to the site – reputation scores and badges – clearly add to the site’s stickiness and appeal, though are not without potential problems, including the tendency for answers to be provided quickly in a rush for the accompanying reputation points.

Truede et als’ study is qualitative, looking at the types of questions that are asked and the tags applied to them. They propose a coding of programming language; framework; environment; domain and non-functional for tag categories and question types including how-tos; discrepancies; environments; errors; decision and conceptual. They found code review questions to be the most frequently answered satisfactorily (ie with an “Accepted” answer in 92% of cases studied), proposing that this is the easiest category to answer being the most concrete and self-contained, usually containing a code snippet for answerers to critique and correct.

Kumar et al looked at the Stack Overflow platform as an example of a “two-sided market”, where there are two distinct user groups – questioners and answerers – and a network effect between them that helps to determine its attractiveness to new adopters. Using a data extract, they derived “attachment curves” for the platform, defined as the probability that a user of one type or another will join the platform. They found a strong “cross-side” network effect, where the rate of questioners joining the network was strongly influenced by the presence of answerers, but that this effect was asymmetric and answerers grew more slowly in the presence of many questioners.

These papers together provide some good insights into how this online community has grown and been a great success – there is a symbiotic and sustainable balance between questioners and providers, groups both being served by the interactions, to the huge benefit of the third group, the search engine visitor. We also see how some question types and categories work more successfully, and how rewards might encourage speed and quantity but may also compromise quality.

REFERENCES

KUMAR, R., LIFTSHITS, Y. and TOMKINS, A., 2010. Evolution of Two-Sided Markets, Third ACM International Conference on Web Search and Data Mining, February 3-6 2010
http://research.yahoo.com/pub/3015

MAMYKINA, L., MANOIM, B., MITTAL, M., HRIPCSAK, G. and HARTMANN, B., 2011. Design lessons from the fastest q&a site in the west, Proceedings of the 2011 annual conference on Human factors in computing systems, 2011, ACM pp2857-2866
http://bid.berkeley.edu/files/papers/mamykina-stackoverflow-chi2011.pdf

TREUDE, C., BARZILAY, O. and STOREY, M.-., 2011. How do programmers ask and answer questions on the web?: NIER track, Software Engineering (ICSE), 2011 33rd International Conference on, 2011, pp804-807

PFEIL, U., SVANGSTU, K., ANG, C.S. and ZAPHIRIS, P., 2011. Social Roles in an Online Support Community for Older People. International Journal of Human-Computer Interaction, 27(4), pp. 323-347

WELSER, H.T., GLEAVE, E., FISCHER, D. and SMITH, M., 2007. Visualizing the signatures of social roles in online discussion groups. Journal of Social Structure, 8

This set of studies looked at data from online support and discussion communities with the aim of understanding network structures and the parts played by types of community members. Cobb et al used network analysis alone in their study of a community of smokers looking to give up (QuitNet). Pfeil et al attempted to integrate content and structural analysis in their analysis of the SeniorNet community for older people to discuss health issues. Welser et al sought to correlate structural characteristics with predominant interaction behaviours in three sets of Usenet data (two technical discussion groups and one on kites!).

Image: Flickr/barabeke

Cobb et al. start out by comparing weakly connected with strongly/densely connected members – on the basis of blog post and private message connections. They find more of the strong or densely connected core are women, and that they are older than the weakly connected core. They are also less likely to have lapsed in their attempts to give up. The study goes on to identify three significant groups: Key Players (50 users of the 800,000 registered and of the 7500 or so active in the study period), Newcomers (800) and Integrators (750). Key players were 80% female, with an average age of 49 and a high level of activity in terms of posts, connections and reciprocal messaging. Newcomers were all smokers (avg age 39) who joined the site during the study. Integrators were those who formed ties with the newcomers by initiating contact. The authors note a healthy heterogeneity to the network, with abstainers mixed with smokers – thereby providing the normative influence needed for quitters to keep it up.

Pfeil et al used a coding scheme to tag forum posts in a much smaller community (only 30 or so active members analysed) according to: self-disclosure, community, factual, slightly-off and support oriented content. The tagging scheme yielded seven types of user based on the predominance of these post types. They also undertook a regular equivalence analysis (REGE algorithm) of the network, which yielded six groups showing distinct interaction patterns. In their discussion they attempt to integrate the two groupings, though note that:

“At first glance, it appears that there is no overt connection between the results of the structural approach (social network analysis) and the behavioral approach (content analysis). The groups identified by the two approaches contain different members. However, at closer look, some relationships can be found.”

The groups they arrive at are:-

• Moderating supporter: Givers and receivers of support, active in welcoming new members and posting frequently at length;
• Central supporter: Frequent posters proving support and strongly connected, but less active in welcoming new members;
• Active member: Regular posters mostly chatting about themselves;
• Passive member: Less central. Did not provide support but did show self-disclosure
• Technical expert: Short messages on technical issues.
• Visitor: Remote positions, limited engagement.

Pfeil et als conclusions – that moderating supporters should be recognised and cultivated – are not especially surprising or new, though they do make an interesting observation about building trust in order to encourage self-disclosure.

Welser et al’s study focussed on the answer person “a participant in an online discussion group whose primary mode of interaction is the provision of helpful, informative responses to other group members’ questions.” Answer people are important for contributing without expectation of recompense and have high community value. For this reason the authors wanted to see what structural aspects of communities correlated well with answer people. They found a characteristic pattern of a sparse, star shaped egocentric network with connections to isolates, few of whom were connected to each other (see their example). Their analysis found that “authorlines” (thread contribution patterns), neighbour’s degree distribution and local network characteristics were good predictors of answer people.

The studies are methodologically interesting, with shortcomings of a purely structural approach noted by Cobb et al – for example, it is impossible to discern the influence of the community on more passive members. And what are the motivations of the key players? Pfeil et al tried to triangulate but found it difficult, suggesting that one-dimensional categorisation of users may be an over simplification.

.

In this paper, Mercier argues that an evolutionary account of reasoning that derives from social argumentation is more compelling than the classic view which frames it as a crown jewel of individual cognition. He sees reasoning as a necessary outcome of human communication, where the responder employs sophisticated coherence checking of newly received facts (tending towards conserving existing knowledge), and the sender employs techniques to select the most convincing premises to persuade the receiver (while striving to avoid an irreversible loss of trust).

Mercier draws support for this theory from the observations that:

1. Confirmation and disconfirmation bias are a common and robust experimental finding, with people finding support for their side of an issue much more readily than the other side;
2. Individual reasoning doesn’t necessarily lead to better decisions, and here Mercier cites evidence for poorer decisions when people are asked to consciously and explicitly consider their reasons first.
3. Individual reasoning tends toward options that are easier to justify, and this may lead people away from normative answers toward those that would be easier to defend if called upon to do so.

Individual reasoning is therefore “a tool of (anticipated) persuasion”. Groups can perform better, on the other hand, perhaps due to a fairer balance between evaluation and persuasion. Here Mercier cites the “truth wins” and “assembly bonus effects”, where groups converge on correct answers, even where no individual members have these answers to begin with. In ideal social argumentation, the net effect can approach unbiased reasoning, because the type of biases named above provide the means to divide labour (each person only needs to find arguments for their own position and these are judged by the rest of the group on their merits).

Mercier does not always explain the evidence that he draws on very clearly, and it is tempting to wonder if he is exhibiting confirmation bias himself. Clearly neither the shortcomings of individual reasoning nor the merits of group decision making should be over-generalised. That said, there seems intuitively (i.e before engaging reason!?) to be much value in his approach in seeking a social, relationist origin to this cognitive capability. These ideas seem to fit well with that of enactive cognition – that individual cognitive structures are a consequence of a structurally coupled social system – and social epistemology, that social evidence should be considered a valid and primary source of knowledge.

Tollefsen, D.P. (2009) WIKIPEDIA and the Epistemology of Testimony. Episteme 6 (1), 8-24.
Wray, K.B. (2009) The Epistemic Cultures of Science and WIKIPEDIA: A Comparison. Episteme 6 (1), 38-51.
Sanger, L.M. (2009) The Fate of Expertise after WIKIPEDIA. Episteme 6 (1), 52-73.
Magnus, P.D. (2009) On Trusting WIKIPEDIA. Episteme 6 (1), 74-90.

In this set of articles, philosophers considered the implications of Wikipedia on theories of testimony, taking various positions on its role and significance as a knowledge source. A common theme is what our use of a collaborative but mostly anonymous system tells us about human credulity. A further thread is how many of the cues we normally use to assess credibility are not available in Wikipedia. Although there still ways to determine how accurate its articles are, we don’t actually tend to use them in practice. In taking the philosophical tack, the authors join up our current knowledge seeking habits with traditions within philosophy, but also remind us of the huge power and reach of Wikipedia.

In the first article, Tollefsen claims that Wikipedia can count as testimony on the assurance principle – that the authors are providing some assurance about the correctness of the content. Moreover, it has many qualities of group testimony as distinct from the aggregation of individual testimonies. She notes that the more well established articles represent a balanced view of the topic (as we’ll see this is perhaps overidealistic and is not entirely corroborated by the other authors.), seen in a way as the settling of the group mind on an issue. As we don’t know the individuals or groups involved in authorship, we are also to some extent placing our trust in a system.

Tollefsen largely supports an ‘antireductionist’ approach to justification, where you you are justified in believing others as long as you have no reason to believe them insincere. In the case of Wikipedia, however, she concludes that default entitlement to trust does not hold, and that we need to treat Wikipedia more cautiously, like “talking to a child”. Later on, she notes that we monitor testimony by checking it against our background beliefs, and this is how we may choose to accept the content of articles.

Wray takes the approach of comparing the process behind wiki articles to the practices of scientists, and comes out more in favour of science as a sound knowledge making practice. He observes how hard it is to authenticate online claims, quoting Shapin that “trust is ubiquitous in knowledge societies”. He further notes that the reliability of wikipedia is based on ‘invisible hand justification’ – in that there is no one person in charge of quality assurance on an article.

Wray sees wikipedians as having less to lose than collaborating scientists and therefore less incentive to be honest and conscientious. That said, he does go on to note that the rush to publish in science is a knowledge anti-pattern and may lead to as many accuracy issues as the rush to post online, which he casts as a form of gossip.

Wray compares egoism and non-egoism as justification in testimony. An egoist approach would say that you need to know the character of the person testifying, and a non-egoist would say it is just enoughto be told someone else believes something. But he concludes that neither extreme really holds in Wikipedia – we dont know the reputation of the authors nor if they really believe what they tell us.

Sanger, a philosopher as well as one of the originators of Wikipedia discusses whether experts still needed now that non-experts can so effectively aggregate knowledge. He concludes, however, that the role of the expert has been too far downplayed in Wikipedia, as experts may be pushed out or ignored.

In conflict with Tollefsen’s views about the “balanced” article emerging over time, Sanger claims that article quality may be a function of the persistance and aggression of the article’s followers, not just the lifetime of the article. In content wars, experts may be driven off by aggressive amateurs with more time on their hands.

Magnus asks what it means to trust Wikipedia as a knowledge source. He notes that in the famous (though controversial) Nature study, Wikipedia proved more variable and slightly less accurate than Brittanica, though perhaps in Wikipedia omissions are more common than inaccuracy.

Magnus provides a useful list of cues that we may use in assessing written testimony including: authority, plausibility of style and content, calibration and sampling with usual sources. He notes that these come undone on Wikipedia. Authority is hard to determine – and not guaranteed with referencing. Style can be edited out. There is a real risk of plausible falsehoods in the content. Calibration may only occur on widely known, as opposed to more specialist portions.

Magnus offers some solutions: Follow sources, interpret the palimpsest – the evolution of the article – from the discuss pages, and link to the dated version of the page. All things that are possible, but even academics probably don’t do.

While the more cautious views on how we “should” treat Wikipedia presented in this journal are coming from experienced academics, I am fairly sure that the position of the average user – including students and teachers – is much more toward anti-egoism and anti-reductionism. Thomas Reid’s principle of credulity – that we are predisposed to trust our fellow man – certainly holds in the large part, especially as often there is not much at stake if we are taking on false beliefs. As the need for correctness increases, however, then so does the need to spend time (and cognitive resources) in assessing credibility.

One area that I felt was not well covered in these papers was the tendency for online information to be recycled. The verifiability principle mentioned by Sanger works well, though often means that large sections are lifted directly from pre-existing sources on the topic, many of which are available online, many of which are written by “experts”. I expect that there is research available on the extent to which Wikipedia articles are largely paraphrased prior texts, and will report it here if I come across it.

In this chapter of the Annual Review, Fallis argues for social epistemology as the best philosophical underpinning to information science. Providing access to society’s knowledge through recorded information is argued as the main reason for information services. Interestingly, social epistemology was first coined in information science (by Shera) , though was also developed as an idea independently within philosophy.

Fallis covers the debates within philosophy as to what constitutes knowledge, and how these have been viewed by information scientists, most of whom have naturally taken more pragmatic perspectives. He also notes the different (classic v revolutionary) views of social epistemology, with the former a social extension of the established idea of knowledge as justified true belief, but the latter taking the social constructivist line. A further useful distinction made is between social epistemology as a normative project — the goal of which is to maximise knowledge acquisition (or at least maximise access to authoritative information) — and the sociology of knowledge, which is about understanding and describing how social factors contribute to knowledge making.

In outlining debates about necessity of the truth condition, Fallis notes that librarians will help the user try to discover the “truth”, which the user in turn sees as the key objective of using the service. That said, it is acknowledged that the collection owner cannot possibly know whether all the facts contained therein are true.

Some interesting and important conclusions come from acknowledging the centrality of social epistemology in this way:

1. That epistemic principles should guide collection organisation (e.g. by type of claim and evidence offered);
2. That information providers should collocate a range of viewpoints on a topic to allow the seeker to better determine the truth (a principle taken from Mill’s On Liberty);
3. That any aspect of an information service that enables the social acquisition of knowledge is as important as any other (sofas and coffee bars are as important as books)- I liked this one!

I felt this to be a very rich and balanced article with many considerations of lasting relevance as we move into the digital age. I also really liked the exploration of the underlying goals of information services, a topic that needs to be foregrounded more often.

This paper argues for a social capital approach to system analysis, based on the failure of KM 1.0 to acknowledge the importance of factors such as community ties and belonging to motivate and stimulate knowledge sharing. Intoducing sociotechnical approaches to design, the authors stress the need to understand “infoculture” (providing the motivation) in addition to infrastructure (opportunity) and infostructure (ability). These correspond to the relation-based, structural and cognitive dimensions of social capital respectively.

Design approaches that concentrate on structural aspects are criticised as technologically deterministic, and more socially determined design approaches are needed to better adapt technology to the real ways that people interact.

Written as it was before the meteoric rise and mainstreaming of social networking and web 2.0, the paper shows its datedness through some of the examples cited. But many of the principles expressed, such as the need for ethnography and pattern recognition are highly relevant today.

The hypothesis closes the paper – that better social alignment of IT will reduce the need for sociotechnical brokers – has probably not been borne out despite the popularising of the social web. Though perhaps these roles are becoming more devolved to “normal users” today.

Flickr - © Big Fat Rat

This study analysed a selection of threads from the Metafilter community blog to determine how it related to previously proposed characteristics of Communities of Practice — namely the nature of knowledge sharing; “Legitimate peripheral participation” (how newbies become accepted); Warrants (relevance) and Identity/Reputation.

They put forward evidence for each of these processes within the community blog. On knowledge, selected excerpts relate to the nature of the initial post (should be fresh and thought provoking) and the quality of the resulting discussion. On warrants and LPP, they note that — despite the system being reasonably open — the old timers having developed methods for discouraging new posters that don’t seem to add value, and encouraging those that do.

One of the interesting points for me was the idea of signal to noise and the fact that one person’s signal is another’s noise — even the old timers seeming to acknowledge that the noise is part of the acceptable baggage of the community.

The authors offer some recommendations on community design, though most of these are not particularly new — ie. the need for the technology to fit the community, not vice versa and the need for moderation in some shape or form. Also, the utility of identifying members (length of membership & contribution) to confer some sort of authority.

This paper presents exploratory research around the topic of wisdom, exploring how it has been defined in the literature – both eastern and western – and what it means to postgraduates of information and library studies and information technology.

In their literature review, Rowley and Slack give a range of conceptions of wisdom, some more dependant than others on having knowledge as a foundation, with most seeing a role for action, judgement and foresight. Wisdom in organisations is also discussed, with the interesting idea of “Kairos time” being mentioned, or the ability to take the right action at the critical moment. Organisational strategy and wisdom is further linked with CSR, in that wise decision making today is made more complex by the ethical and social considerations of a globalized and networked world.

In a simple exploratory research design, the authors asked postgraduates to complete the sentence “wisdom is..”, then undertook content analysis on the responses to draw out key elements of the concept as understood by that population. Many – but interestingly far from all – conceptions included knowledge or knowing, experience and a sense of action or application.

At first I found it difficult to see the value of this work beyond the core concept analysis, which though interesting in its own right, didn’t seem all that useful. But I think it is in the relation to organisational learning that this work can be interestingly extended, through the characterisation of “wise” decision making and perhaps though contrasting the conceptions given here here to newer models such as crowd wisdom on the social web.

Blackler compares more traditional organisational learning (KM) approaches to knowledge to newer theories of knowing and comes out rather in favour of the latter.

He starts by discussing theories that see knowledge as some version of : encultured, embrained, embodied, embedded or encoded. He argues that knowledge is seen as either belonging to people (through personal knowledge or know-how), social groups and processes or being encoded in symbols and that each of these is needlessly reductionist and cannot stand alone. He goes on to relate these to the knowledge economy and types of organisations, noting a shift toward embrained (knowledge intensive) and encultured (communication – intensive) types.

The discussion also covers the transformative nature of technology, not unlike earlier technological revolutions. Just as writing seemed to “kill living eloquence”, Blackler notes how ICT disrupts the traditional significance of traditionally valued types of knowledge. In a quite graphic metaphor, he states how technology is leading organisations to simultaneously implode into electronic codes and to explode into global networks.

The paper quotes reviews of theory emerging from anthropology, ethnomethodology and activity theory, and notes that knowledge is always constructed and transformed in use. Cognitions are likewise situated and collective. As such, the author thinks it is better to talk about knowing as a multidimensional phenomenon.

Blackler sees activity theory as a potentially useful in understanding this version of knowledge. The theory sees knowledge as dynamically interconnected and constructed from social interaction and practical collaboration. The work of Orr and Englestrom are quoted. Englestrom’s model of socially distributed activity systems are discussed :

Engestrom’s distributed activity system

The triangles represent the tensions and interactions that the individual deals with, and learns to cope with.

Blackler concludes that it is not helpful to focus on what types of knowledge are needed, but on how systems and activities are evolving to and changing as a result of globalisation and the march of technology. Interestingly, the communal narrative aspect of expanded, distributed activity systems, has relevance today in the uptake of social technologies. In this respect Blackler has shown a measure of foresight.

Image:Mary Hodder, Flickr

My attention has just been drawn to the idea of distributed cognition, and in this paper, Yvonne Rogers introduces it as a multidisciplinary, unifying approach to understanding how people and media interact. The argument goes that it is not useful to abstract user and task as classic HCI theory does, but better to study  the system in situ as a whole. No one person or program has the full picture or full control, but together they achieve complex goals.  In this respect, the tech components can be seen to have a type of cognition, in that they embody “representational states”.

I was not wholly convinced by the use of the case studies in this paper to describe the approach (a technical support team and a hospital system), but will be following up the work of Hutchins in the early – mid 90′s, who first came up with the idea.