Todo-List for Harvest

Short Term Goals

• Port query parser from old search.cgi to zquery.pl.
• Evaluate either to normalize or drop relativ links in summaries.
• Finish implementation of "Sh" indexer.
• zquery.pl
  • Evaluate ranking.
  • Implement proximity search.
  • Implement attribute (element) search.
  • Implement load limiter/query cost guesser.
  • Implement load balancing with replicas.
  • Consider how to deal with very expensive queries like searching for stop words (the, and, etc.).
  • Implement logging (term, time, result).
  • Evaluate stopword elimination.
  • Remove unused functions read_hash_zoomsh() and remove_quirks().
  • Create internationalization framework.
• soif2xml.pl
  • Add optimization flags (-O0, -O1, -O2, etc.) for different levels of summary tuning. -O0 does nothing, -O1 removes multiple whitespaces, -O2 removes obvious junk like "-----" -O3 does line reformatting etc.
  • Reformat summaries to have approximately 80 characters per line? (fmt -w 80 or use Perl's format and write) This should make parsing results easier and faster, but we have to evaluate if it's worth the additional work to reformat every object in the gatherer if it turns out, that the number of hits and displayed summaries involves just a small fraction of the numbers of objects in broker.
  • Truncate length of element to a certain maximum of for example 20KB and add a word list (element "keywords")? This will affect ranking.
  • Normalize the element set? We are currently working with well formed XML files. Would it be useful to use valid XML files? Do we need XML DTD or Schema?
  • Add additional element, for example <result-display> for displaying results? This should minimize work for zquery.pl.
  • Put <url-references> back into the xml files, so we can do "link: http://www.my.do.main/" to get a list of documents pointing to the given document. It should require normalization and duplicate elimination of the url-references.
  • Move the functionality of soif2xml.pl into broker.
• Modify essence not to add <description> tag into summaries (summarize.c, mkdescription()) and not to do any summary normalization.
• Create a query interface which enables to use the rich features of Zebra, but is not too complicated for casual users.
• Build an XSL for Harvest's XML files?
• Create/evaluate XSL for presentation of search results, which are in XML.
• Add possibility to keep the raw HTML pages and not to delete them after summarizing, so we can evaluate the summarizers. This should also make it easier to plug in other gatherer like larbin.
• Write object analyzer to generate/analyze stop word list, word frequencies, size of documents, list of possibly misspelled words. Try to generated a list of similar words to suggest for queries with no hits.
• Evaluate the viability of creating and using thesaurus for searching
• Tighten permission for files and directories in broker directory.
• Make $HARVEST_HOME/RunHarvest, $HARVEST_HOME/bin/CreateBroker and $HARVEST_HOME/bin/Harvest obsolete. Creating broker should be a matter of copying template files and editing a configuration file.
• Simplify the management of multiple gather daemons (gatherd). Currently you have to assign a different port for each gatherds and have to add them as collection points in brokers.
• Automate zebrasrv restarts and index updates.
• Do some research on encoding and language recognition, evaluate the possibility to use iconv to build a collection of UTF-8 or other suitably encoded full-text material.
• Evaluate the impact of filtering out stopwords from summaries, do some semantic check to filter out search irrelevant data like words longer than a certain number of letters, repeating letters more than n times, mixed letter and digits, words containing only digits, words containing arbitrary mixed case letters, etc.
• Reorganize installation process so that zebra binaries are installed in lib/broker and curl installed in lib/gatherer.
• Add support for embed tags in gatherer, so that we can summarize Macromedia Flash files.
• Evaluate qdbm's full text engine.
• Drop support for Glimpse in 1.9.
  • Remove URI element from summaries.
• Port fixes to 1.8 series.
  • xlhtml fix.
  • quicksum.cf fix.
  • summarize.c fix.
  • Remove md5 based duplicate elimination.
  • Merge Russian translation of Harvest User's Manual.
  • broker.conf.in typo fix.
  • Port changes around brokerAdmin.cgi.
  • Fix links in download.html.
  • Fix gzip download link in gather.c.
  • Evaluate/implement result grouping in search.cgi.

Gatherer

• Add hooks to plug in a customized candidate selection filter. This would make it possible to use for example some statistic based methods to distinguish between useful pages and junk.
• Implement candidate selection filter for httpenum.
• Modify httpenum to trust mime type sent by http servers.
• Fix crashing httpenum when essence dies.
• Fix crashing httpenum when database is not in sane state.
• Fix essence not to crash when receiving a signal.
• Unify breadth and depth first enumerator.
• Emulate breadth first enumeration for HTTP-If-Modified-Since gathering for httpenum-breadth.
• Make Delay=x work when local disc cache is disabled. Currently only global "Access-Delay" works. Possibly a bug in enum.c not to call set_envs() for child process.
• Check performance with local disc cache switched off and disable it, if it doesn't slow down gathering.
• Modify Gatherer to spawn multiple enumerators where possible, e.g. the hostname is uniq, hostmax = 1, and the number of running enumerators is smaller than a defined limit. This will be a prototype only, because the http enumerator is too heavyweight 25MB+, to make it really usable.
• Instead of creating temporary filename with tempnam() and keep track of temporary files to delete after processing, use a single name for temporary files and let them be overwritten by the next item. (Wait with this until the implementation of spawning multiple gatherer, since it might be useful when multiple gatherer creates temporary files in the working directory.)
• Spawn multiple enumerators to multiple hosts when possible.
• Add support for importing data from htdig 3.2.
• Add support for importing data from larbin, webbase, aspseek, mnogosearch.
• Add HTTPS support.

Broker

• Store objects as QDBM BLOBS.
• Store book-keeping information apart from fulltext data.

Documentation

• Switch from linuxdoc to docbook.
• Document environment variables.
• Document how to plug in other gatherers like htdig.

General Issues

• Make fulltext objects shareable between gatherer and broker when they are running on same machine.
• Get rid of LeafNode option?

Home | Sites using Harvest | Download | Contributed Code | Todo List | Links Contributors | User's Manual | FAQ | Installation | ChangeLog | NEWS