Technologists could yet prove best friends to journalists

WIRED: MANY YEARS ago, back when we did not despise anyone in thefinancial industry on sight, I heard a description of a workingpartnership at a Wall Street firm that sounded almost romantic.

A party guest told me how he worked in a firm where individualtraders would work paired with a programmer colleague, who would sitbeside his companion, nestled together in front of the banks ofscreens and terminals.

While one would do the trading, the other would, on the fly,construct entirely original programs to match and augment thattrader's instincts. The coder would work to create new ways tovisualise the market, and automatically mine for new information andpatterns in the endless data flow of stocks.

At the time, I rather liked the idea of two intrepid explorersfabricating the tools they needed to pan for gold in a newinformation frontier. I like the latest version of this pairingmore.

Instead of tying financial whizz kids with coders, this newpartnership connects investigative reporters to the same kind ofdata-miners. Instead of pure profit, these couples hunt for stories:tales of corruption, graft, and hidden motives, buried in themountain of data that corporations, governments and individuals emitin the modern age.

Most reporters can't afford to hire a side-kick coder, of course -especially in a world where technologists seem to have emptiedjournalism of many of its traditional revenue streams. The patternthat emerges in this new media is more of a hybrid.

Some of these data journalists are reporters who have learned toprogram themselves. Others are programmers who have chosen a careerchange as journalists.

Either way, large forces in the traditional world of journalismare betting that technologists and reporters work together in thefuture more than they work against each other.

The Knight Foundation is one of the biggest non-profit funders ofthe cutting edge of journalism. Based in the cities where USnewspaper moguls the Knight brothers ran their papers, it has assetson close to $2 billion. For the last five years, it has handed out atotal of $27 million to those experimenting in the building of toolsfor new forms of journalism in those cities in a competitive grantprocess called the Knight News Challenge.

More than 12,000 projects applied, but only 76 were funded. Thelast 16 were announced this week.

The winners this year continue the trend of trying to graftjournalist and programmer together: or hackers and hacks, as oneglobal meet-up between the two cultures calls it. Some awards wereaimed at providing flexible tools and number-crunching forgrassroots, citizen reporting.

Half a million dollars went to the Public Laboratory for OpenTechnology, which famously helped residents generate high-resolution "satellite maps" to chart the local extent of the Gulfoil spill, using helium balloons and digital cameras.

Others are aimed at providing assistance for professionaljournalists, including $475,000 to the Associated Press's Overviewproject which will refine, mine and display large data sets fromgovernments and other sources.

Knight has also been funding individuals, including a scholarshipprogramme to retrain coders as journalists. Brian Boyer was a formerKnight scholar who went on to win a $150,000 grant for his Pandaproject, a set of open source web tools his team is building to helpreporters seek out patterns in data using the web's alreadyextensive array of analysis tools.

This new move to "hybrid data journalism" is certainly a searchfor how to do journalism in an internet world but, unlike those WallStreet profit-seeking duos, I don't think it's a question thatrevolves exclusively around finding the money. It's more aboutworking out how to do journalism better.

We live now in an environment where the lowest hurdles ofjournalism - expressing an opinion, doing some research, uncoveringa new story - are made far easier for anyone to surmount. I canpublish an opinion with a few clicks online. The research tools theaverage smart surfer has dwarf the most expensive databases hoardedby media companies 10 years ago. Hoards of online detectives cantrack their obsessions far better than the hired hand of a localreporter on their beats.

What remains difficult is filtering, scanning and picking outnovel signals: the far-out stuff that the net, en masse, would neverfind on its own.

And I think that's actually closer to the real reason why thosetraders teamed up with coders. It wasn't about money. It was aboutcompetition. In order to get ahead, financial wizards worked notonly to be smarter than the next person, but to be smarter than themarket as a whole. The money emerged, in theory at least, from that.

There are dangers, clearly. What felled the financial industry isthat, in the end, they were all pursuing the same goals, listeningto the same signals, deriving the same conclusions. There's often anelement of groupthink among geeks, just as with journalists, and itwould be a tragedy if the two were to combine to form onemonoculture, chasing its own tail.

But if journalists really are in pursuit of novelty andinnovation, technologists might be their best friend in creating thetools that might let them find it. To succeed, those tools need tobe distributed widely, with no preconceptions about how they shouldbe used, or who should use them.

The stories of the future are out there: but the further awaythey are from the stories we tell in the present, the moreresilient, and more honest to the real world, they will be.