Implement candidate selection filter for httpenum.
Modify httpenum to trust mime type sent by http servers.
Fix crashing httpenum when database is not in sane state.
Fix essence not to crash when receiving a signal.
Unify breadth and depth first enumerator.
Emulate breadth first enumeration for HTTP-If-Modified-Since
gathering for httpenum-breadth.
Make Delay=x work when local disc cache is disabled. Currently
only global "Access-Delay" works. Possibly a bug in enum.c not to
call set_envs() for child process.
Check performance with local disc cache switched off and disable
it, if it doesn't slow down gathering.
Modify Gatherer to spawn multiple enumerators where possible,
e.g. the hostname is uniq, hostmax = 1, and the number of running
enumerators is smaller than a defined limit.
This will be a prototype only, because the http enumerator is too
heavyweight 25MB+, to make it really usable.
Instead of creating temporary filename with tempnam() and keep
track of temporary files to delete after processing, use a single
name for temporary files and let them be overwritten by the next
item. (Wait with this until the implementation of spawning
multiple gatherer, since it might be useful when multiple gatherer
creates temporary files in the working directory.)
Spawn multiple enumerators to multiple hosts if possible.
Add support for importing data from htdig 3.2.
Add support for importing data from larbin, webbase, aspseek,
mnogosearch.
Add HTTPS support.
Broker
Minimize size of Registry.
Write shell-client for broker so we don't have to compile in the
indexing method, but call a shell script to manage the indexer, e.g.
instead of calling glimpseindex directly from the broker, the broker
would call build_index.sh, which builds an index with an arbitrary
fulltext engine.
Separate data from metadata on disk.
Documentation
Switch from linuxdoc to docbook if it is easier to create xref and
appendix with docbook.
Build some examples for *.unnest.
Document environment variables.
Document how to plug in other gatherers like htdig.
General Issues
Make SOIF objects shareable if gatherer and broker run on same
machine.