Hypermail

One of the first integrations of the Web with email, Hypermail turns standard mailing lists into published Web accounts and institutional records of conversations. This allows the product of collaboration, the email conversations where people hash out ideas and make decisions, to be a source of knowledge.

In the early days of the Web, a small number of people used email conversations to work out many of the core decisions that would make the Web a success. Hypermail captured these conversations so others could catch up, learn, and contribute. It’s still used in the public archives of the W3C standards organization. Hypermail and its progeny such as Pipermail continue to drive the publication of tens of millions of mailing list conversations on the Web.

Beyond the fundamental idea of turning conversations into institutional memories, Hypermail’s innovations included thread and link finding algorithms, which can be found in most modern mail and threaded discussion systems today. It established the convention that each message should have its own URL, so that one could discover and link to the message itself as archival content and follow links to related pages. For example, here is a link to an early message from Tim Berners-Lee, pleading with key people to work together on what would turn out to be key Web browser ideas. Hypermail inferred next-in-thread links by analyzing the email headers and content, so that the conversation can be followed from Sir Tim to Dan Connolly to Marc Andreessen and other web pioneers.

Hypermail was a collaboration between Tom and Kevin Hughes, an early Web pioneer. Tom conceived of the ideas, analyzed a corpus of sample email data, wrote the algorithms, and implemented the first working version in Lisp in 1994. It was deployed on some of the key mailing list servers of the day. The work was paid for by an early Internet company (EIT) as a fun project. Even though it was meant to be free, the sponsor’s legal advice required people to sign something before using it. Kevin, a colleague at EIT, saw the future. He convinced the “powers that be” that this thing needed to be free, in the sense of open-source code that was easy to download and deploy. Kevin rewrote the code in C and released it without restriction to the public, and after that it took off. It is still distributed freely, and has an open-source community maintaining it.

A Technology for Collective Memory

At the time, it was surprising how powerful something as simple as Hypermail could be as a technology to create collective memory.

The essential elements are (1) a work process that people already do, which generates content in context; (2) a mechanism to organize the content that preserves and reifies the context; and (3) a retrieval mechanism that allows people who were not part of the original work to learn from those who were. For Hypermail, the work process was collaboration over email lists. Hypermail organized the content to reify the context of conversations and threads, and the earliest search engines on the web provided the retrieval mechanism. With these elements in place, someone who was not on the original mailing lists could find conversations relevant to their needs, learn from those conversations, and meet potential collaborators.

The power of this simple instance of collective memory was an influence for Tom to create a company for collective intelligence: an enterprise-scale group memory. The company and product, called Intraspect, mediated conversations happening in email and Web-based collaboration tools to capture the knowledge created in collaborative online work. With special search tools tuned for enterprise content, Intraspect enabled people to quickly learn from anyone in their virtual organization. That innovation scaled to organizations of tens of thousands of knowledge workers around the world, spanning traditional company boundaries.