Reply to this post | Go Back

View Post [edit]

Poster:	docdtv	Date:	Jan 30, 2005 2:09am
Forum:	web	Subject:	Full-text search of the Wayback archive

What happened to this experimental tool? Was it withdrawn because it used up too many computer cycles? If this was the problem, could a scheme using a peer swap of compuer power (a la the SETI project) "finance" compute-intensive tasks?
This post was modified by docdtv on 2005-01-30 10:09:38

Reply to this post

Reply [edit]

Poster:	Brak	Date:	Jan 30, 2005 2:31am
Forum:	web	Subject:	Re: Full-text search of the Wayback archive

Now that's a fascinating idea. It's not the reason it was taken down. We hope to have that functionality available again.

Doing a distributed index sounds like a worthy p2p project for all sorts of reasons. The problem with projects of this scale might be the sheer size of the data. If the index were, let's say, 25 Terabytes, then if each user donated 20GB of hard drive space, it would take, in theory, a minimum of 1250 users, and more realistically 5000 users (taking into account redundancy and load sharing). That's a taller order than most p2p apps. Anything is possible, however.

For now, stay tuned...