Archive for the ‘info-overload’ Category

The other America

Friday, July 23rd, 2010

The only sign most people have that they’re passing the headquarters of the world’s largest spy agency is that their GPS

stops working.


It’s with such evocative details that the Washington Post paints its portrait of Top Secret America, an “alternative geography of the United States” that took more than two years of research to assemble from public records. Parts of the project are so crowded with Byzantine bureaucracies, mysterious devices and vast sums of unaccounted money that they read like mediaeval travellers’ tales; others focus on the quotidian, from casino-themed networking nights for employees with Top Secret clearance to the Director of Counterterrorism’s battle to read all his email on the same computer.

It’s perhaps appropriate that this project has, like the maneuvering rival-allies in the cozily named US Intelligence Community, its own double across disciplinary lines — Trevor Paglen has spent eight years exploring and documenting the “black world” in a series of books and exhibitions (the image above is from his Symbology project on the insignia of classified military units). But whereas the Post is hungry for facts, to the point that its investigation ironically suffers from the same information overload it diagnoses in the intelligence world, Paglen’s lens is always focussed on the point where certainty ends and secrecy begins: the holes in the map that modernity promised to sew shut, and that its left hand is now busily unpicking as quickly as its right can close them.

If we had thought our networks, processors and databases would push back the frontiers of ignorance, we were right; what we couldn’t have known was that the territory would be populated by chimeras: that which we know but can’t reveal, that which we know but can’t find, and that which we know but can’t understand.

Intelligence-gathering by sneakernet

Tuesday, January 5th, 2010

A new report by senior US intelligence officers recommends sweeping changes to intelligence-gathering practices in Afghanistan. The two most interesting recommendations:

  • Intelligence work should be divided along geographic, rather than functional, lines. “The alternative – having all analysts study an entire province or region through the lens of a narrow, functional line (e.g. one analyst covers governance, another studies narcotics trafficking, a third looks at insurgent networks, etc) – isn’t working.” (p4)
  • Analysts should aggregate intelligence by regularly travelling to visit those who collect it. “Information essential to the successful conduct of a counterinsurgency is ripe for retrieval, but analysts that remain confined to restricted-access buildings in Kabul or on Bagram and Kandahar Airfields cannot access it.” (p17) The internet is not suitable for this purpose because “vital information piles up in obscure SharePoint sites, inaccessible hard drives, and other digital junkyards.”

The first point interests me because it suggests that problem-solving doesn’t always scale through specialisation, as tends to be assumed in academia: when the flow of information is constricted, a geographically-organised hierarchy of generalists may be more effective than a taxonomically-organised hierarchy of specialists.

The second point bears more directly on mobblog’s research interests (though I’m not suggesting we should design communication systems for the US military): manual aggregation and curation of information are still necessary, even when that information is in digital form. More surprisingly, the oldest method of aggregation – sneakernet – remains the most reliable.

The issues discussed in the report might seem specific to the chaotic and poorly connected environment of Afghanistan, but I want to argue that the fundamental problem – finding relevant information in a shifting sea of circumstances, practices, organisational structures and data formats – exists everywhere, and is not solved by better connectivity, nor by making everything digital.

David Weinberger has suggested that in the digital realm, tags will replace taxonomies and it will no longer be necessary to separate the organisation of information from its retrieval; but while the notion of a ‘hierarchy of generalists’ does cast doubt on the usefulness of a priori taxonomies, the recommendation of manual data collection and curation is directly opposed to Weinberger’s ‘tag soup’ approach.

Does this simply reflect a lack of tools (or, God help us, standards), or is the complexity of real-world information as irreducible to tags as it is to taxonomies? Anyone who’s used Google Images will recognise the difficulty of applying tags to non-textual data; assuming the sea never stops shifting, will the extraction of relevant knowledge from information always be a matter of – well – intelligence?

Deconstructing “the Twitter revolution”

Tuesday, July 7th, 2009

Hamid Tehrani of Global Voices gives a sober assessment of the role of Twitter in the Iranian election protests. One of the issues he raises is the temptation to relay breaking news without verifying it. The open source Ushahidi project, which was initially developed to aggregate and map reports of violence following the Kenyan elections in 2007/8, has proposed crowdsourced filtering to deal with this problem. However, the question remains, how can the people aggregating and filtering first-hand reports determine what’s true? Does citizen journalism still require a layer of professional editors, experts and fact-checkers, or can all these functions be shared among the crowd?

Discussing the Netflix Prize

Tuesday, June 30th, 2009

After my last blog post, I was contacted by a journalist who wanted to discuss the effects of the Netflix prize. It seems that now that the competition is winding to an end, one of the real questions that emerges is whether it was worth it. Below, I’m pasting part of my side of the dialogue; other blogs are posting similar discussions, and I’m curious as to what any of you fellow researchers may have to say.



Wednesday, June 17th, 2009

Ushahidi (blog) is an open source platform for collecting, visualizing, and distributing information related to a crisis or ongoing public problem, such as swine flu, election fraud,  and political violence:

Plus, there is also the OMC – it is all about open source mobile phone software, with a focus on humanitarian needs.

Research is the New Music

Monday, February 23rd, 2009

I’ve started trying out a new service, called Mendeley. The quickest way to describe it is a “ for research;” they have a desktop client that can monitor the pdf files that you are reading, and an online presence where each user has a profile. (Read about them on their blog; my profile is here). So far, it seems that they are at a very early stage. However, the basic functionality (seeing/tagging/searching papers you read) seems quite nice. On the other hand, an obvious difficulty is that of extracting accurate meta-data from research pdf files.

The similarity between research papers and songs is quite striking. Think of it this way: songs (research papers) are made by musicians (authored by researchers), have a name (title), and are collected in albums (journals/conference proceedings). Both have a time of release; both can be tagged/described/loved/hated; both are blogged and talked about. Sometimes artists make music videos, sometimes researchers make presentations or demos. (more…)

A Pitch on Future Recommender Systems

Thursday, November 27th, 2008

Yesterday I attended a workshop that was aimed at fostering research collaboration between our department and BSkyB. After a short introduction by the head of the department, a number of members of staff gave short (10 minute) pitches about their past and current research, and areas they are interested in for potential collaboration. The range of work being done in the department is huge- perhaps this deserves a post of its own.


Information Overload

Tuesday, February 26th, 2008

Many of us are researching ways of reducing information overload. The next The Economist Oxford-style debate revolves around information overload. The proposition: “This house believes that if the promise of technology is to simplify our lives, it is failing.” The pro speaker: Richard Szafranski (Toffler Associates). The con: John Maeda (MIT). (more…)