6. Programs and layout of the installed Harvest software

6.1 $HARVEST_HOME

The top directory of where you installed Harvest is known as $HARVEST_HOME. By default, $HARVEST_HOME is /usr/local/harvest. The following files and directories are located in $HARVEST_HOME:

        RunHarvest*         brokers/            gatherers/          tmp/
        bin/                cgi-bin/            lib/

RunHarvest is the script used to create and run Harvest servers (see Section Starting up the system: RunHarvest and related commands). RunHarvest has the same command line syntax as Harvest.

6.2 $HARVEST_HOME/bin

The $HARVEST_HOME/bin directory only contains programs that users would normally run directly. All other programs (e.g., individual summarizers for the Gatherer) as well as Perl library code are in the lib directory. The bin directory contains the following programs:

CreateBroker

Creates a Broker.

Usage: CreateBroker [skeleton-tree [destination]]

Gatherer

Main user interface to the Gatherer. This program is run by the RunGatherer script found in a Gatherer's directory.

Usage: Gatherer [-manual|-export|-debug] file.cf

Harvest

The program used by RunHarvest to create and run Harvest servers as per the user's description.

Usage: Harvest [flags]

where flags can be any of the following:

        -novice         Simplest Q&A.  Mostly uses the defaults.
        -glimpse        Use Glimpse for the Broker. (default)
        -swish          Use Swish for the Broker.
        -wais           Use WAIS for the Broker.
        -dumbtty        Dumb TTY mode.
        -debug          Debug mode.
        -dont-run       Don't run the Broker or the Gatherer.
        -fake           Doesn't build the Harvest servers.
        -protect        Don't change the umask.

broker

The Broker program. This program is run by the RunBroker script found in a Broker's directory. Logs messages to both broker.out and to admin/LOG.

Usage: broker [broker.conf file] [-nocol]

gather

The client interface to the Gatherer.

Usage: gather [-info] [-nocompress] host port [timestamp]

6.3 $HARVEST_HOME/brokers

The $HARVEST_HOME/brokers directory contains images and logos in images directory, some basic tutorial HTML pages, and the skeleton files that CreateBroker uses to construct new Brokers. You can change the default values in these created Brokers by editing the files in skeleton.

6.4 $HARVEST_HOME/cgi-bin

The $HARVEST_HOME/cgi-bin directory contains the programs needed for the WWW interface to the Broker (described in Section CGI programs) and configuration files for search.cgi in lib directory.

6.5 $HARVEST_HOME/gatherers

The $HARVEST_HOME/gatherers directory contains example Gatherers discussed in Section Gatherer Examples. RunHarvest, by default, will create the new Gatherer in this directory.

6.6 $HARVEST_HOME/lib

The $HARVEST_HOME/lib directory contains number of Perl library routines and other programs needed by various parts of Harvest, as follows:

chat2.pl, ftp.pl, socket.ph

Perl libraries used to communicate with remote FTP servers.

dateconv.pl, lsparse.pl, timelocal.pl

Perl libraries used to parse ls output.

ftpget

Program used to retrieve files and directories from FTP servers.

Usage: ftpget [-htmlify] localfile hostname filename A,I username password

gopherget.pl

Perl program used to retrieve files and menus from Gopher servers.

Usage: gopherget.pl localfile hostname port command

harvest-check.pl

Perl program to check whether gatherers and brokers are up.

Usage: harvest-check.pl [-v]

md5

Program used to compute MD5 checksums.

Usage: md5 file [...]

newsget.pl

Perl program used to retrieve USENET articles and group summaries from NNTP servers.

Usage: newsget.pl localfile news-URL

soif.pl, soif-mem-efficient.pl

Perl library used to process SOIF.

urlget

Program used to retrieve a URL.

Usage: urlget URL

urlpurge

Program to purge the local disk URL cache used by urlget and the Gatherer.

Usage: urlpurge

6.7 $HARVEST_HOME/lib/broker

The $HARVEST_HOME/lib/broker directory contains the search and index programs needed by the Broker, plus several utility programs needed for Broker administration, as follows:

BrokerRestart

This program will issue a restart command to a broker.

Usage: BrokerRestart [-password passwd] host port

brkclient

Client interface to the broker. Can be used to send queries or administrative commands to a broker.

Usage: brkclient hostname port command-string

dumpregistry

Prints the Broker's Registry file in a human-readable format.

Usage: dumpregistry [-count] [BrokerDirectory]

agrep, glimpse, glimpseindex, glimpseindex.bin, glimpseserver

The Glimpse indexing and search system as described in Section The Broker.

swish

The Swish indexing and search program as an alternative to Glimpse.

info-to-html.pl, mkbrokerstats.pl

Perl programs used to generate Broker statistics and to create stats.html.

Usage: gather -info host port | info-to-html.pl > host.port.html

Usage: mkbrokerstats.pl broker-dir > stats.html

6.8 $HARVEST_HOME/lib/gatherer

The $HARVEST_HOME/lib/gatherer directory contains the default summarizers described in Section Extracting data for indexing: The Essence summarizing subsystem, plus various utility programs needed by the summarizers and the Gatherer, as follows:

URL-filter-default

Default URL filter as described in Section RootNode specifications.

bycontent.cf, byname.cf, byurl.cf, magic, stoplist.cf, quick-sum.cf

Essence configuration files as described in Section Customizing the type recognition, candidate selection, presentation unnesting, and summarizing steps.

*.sum

Essence summarizers as discussed in Section Extracting data for indexing: The Essence summarizing subsystem.

HTML-sum.pl

Alternative HTML summarizer written in Perl.

HTMLurls

Program to extract URLs from a HTML file.

Usage: HTMLurls [--base-url url] filename

catdoc, xls2csv, catdoc-lib

Programs and files used by Microsoft Word summarizer.

dvi2tty, print-c-comments, ps2txt, ps2txt-2.1, pstext, skim

Programs used by various summarizers.

gifinfo

Program to support summarizers.

l2h

Program used by TeX summarizer.

rast, smgls, sgmlsasp, sgmls-lib

Programs and files used by SGML summarizer.

rtf2html

Program used by RTF summarizer.

wp2x, wp2x.sh, wp2x-lib

Programs and files used by WordPerfect summarizer.

hexbin, unshar, uudecode

Programs used to unnest nested objects.

cksoif

Programs used to check the validity of a SOIF stream (e.g., to ensure that there is not parsing errors).

Usage: cksoif < INPUT.soif

cleandb, consoldb, expiredb, folddb, mergedb, mkgathererstats.pl, mkindex, rmbinary

Programs used to prepare a Gatherer's database to be exported by gatherd.

cleandb ensures that all SOIF objects are valid, and deletes any that are not;

consoldb will consolidate n GDBM database files into a single GDBM database file;

expiredb deletes any SOIF objects that are no longer valid as defined by its Time-to-Live attribute;

folddb runs all of the operations needed to prepare the Gatherer's database for export by gatherd;

mergedb consolidates GDBM files as described in Section Incorporating manually generated information into a Gatherer;

mkgathererstats.pl generates the INFO.soif statistics file;

mkindex generates the cache of timestamps; and

rmbinary removes binary data from a GDBM database.

enum, prepurls, staturl

Programs used by Gatherer to perform the RootNode and LeafNode enumeration for the Gatherer as described in Section RootNode specifications.

enum performs a RootNode enumeration on the given URLs;

prepurls is a wrapper program used to pipe Gatherer and essence together;

staturl retrieves LeafNode URLs to determine if the URL has been modified or not.

fileenum, ftpenum, ftpenum.pl, gopherenum-*, httpenum-*, newsenum

Programs used by enum to perform protocol specific enumeration.

fileenum performs a RootNode enumeration on ``file'' URLs;

ftpenum calls ftpenum.pl to perform a RootNode enumeration on ``ftp'' URLs;

gopherenum-breadth performs a breadth first RootNode enumeration on ``gopher'' URLs;

gopherenum-depth performs a depth first RootNode enumeration on ``gopher'' URLs;

httpenum-breadth performs a breadth first RootNode enumeration on ``http'' URLs;

httpenum-depth performs a depth first RootNode enumeration on ``http'' URLs;

newsenum performs a RootNode enumeration on ``news'' URLs;

essence

The Essence content extraction system as described in Section Customizing the type recognition, candidate selection, presentation unnesting, and summarizing steps.

Usage: essence [options] -f input-URLs or essence [options] URL ...

where options are:

        --dbdir directory       Directory to place database
        --full-text             Use entire file instead of summarizing
        --gatherer-host         Gatherer-Host value
        --gatherer-name         Gatherer-Name value
        --gatherer-version      Gatherer-Version value
        --help                  Print usage information
        --libdir directory      Directory to place configuration files
        --log logfile           Name of the file to log messages to
        --max-deletions n       Number of GDBM deletions before reorganization
        --minimal-bookkeeping   Generates a minimal amount of bookkeeping attrs
        --no-access             Do not read contents of objects
        --no-keywords           Do not automatically generate keywords
        --allowlist filename    File with list of types to allow
        --stoplist filename     File with list of types to remove
        --tmpdir directory      Name of directory to use for temporary files
        --type-only             Only type data; do not summarize objects
        --verbose               Verbose output
        --version               Version information

print-attr

Reads in a SOIF stream from stdin and prints the data associated with the given attribute to stdout.

Usage: cat SOIF-file | print-attr Attribute

gatherd, in.gatherd

Daemons that exports the Gatherer's database. in.gatherd is used to run this daemon from inetd.

Usage: gatherd [-db | -index | -log | -zip | -cf file] [-dir dir] port

Usage: in.gatherd [-db | -index | -log | -zip | -cf file] [-dir dir]

gdbmutil

Program to perform various operations on a GDBM database.

Usage: gdbmutil consolidate [-d | -D] master-file file [file ...]
Usage: gdbmutil delete file key
Usage: gdbmutil dump file
Usage: gdbmutil fetch file key
Usage: gdbmutil keys file
Usage: gdbmutil print [-gatherd] file
Usage: gdbmutil reorganize file
Usage: gdbmutil restore file
Usage: gdbmutil sort file
Usage: gdbmutil stats file
Usage: gdbmutil store file key < data

mktemplate

Program to generate valid SOIF based on a more easily editable SOIF-like format (e.g., SOIF without the byte counts).

Usage: mktemplate < INPUT.txt > OUTPUT.soif

quick-sum

Simple Perl program to emulate Essence's quick-sum.cf processing for those who cannot compile Essence with the corresponding C code.

template2db

Converts a stream of SOIF objects (from stdin or given files) into a GDBM database.

Usage: template2db database [tmpl tmpl...]

wrapit

Wraps the data from stdin into a SOIF attribute-value pair with a byte count. Used by Essence summarizers to easily generate SOIf.

Usage: wrapit [Attribute]

kill-gatherd

Script to kill gatherd process.

6.9 $HARVEST_HOME/tmp

The $HARVEST_HOME/tmp directory is used by search.cgi to store search result pages.