NeatSeeker User's and Developer's Guide

A. NeatSeeker Indexer-HOWTO

The following document gives a brief description of the tasks involved in developing a new type of indexer for NeatSeeker. The document is also available in the project documentation page.

================================================================
HOW TO WRITE A NEATSEEKER INDEXER

Author: Sami Lempinen (lempinen@iki.fi)
$Id: Indexer-HOWTO.txt,v 1.3 2000/10/07 14:24:32 lempinen Exp $
================================================================

1. Introduction

The easiest way to develop an indexer for arbitrary material (text,
XML etc.) is to inherit lempinen.neatseeker.core.AbstractIndexer. It
has a number of convenience functions already defined, and leaves only
the details up to you.

Writing an indexer requires that you implement the
lempinen.neatseeker.core.Indexer interface. If you bypass
AbstractIndexer, you need to implement the methods defined in the
interface yourself. If you inherit AbstractIndexer, you only need to
implement

	public void init(Configuration c)

	       and

	public void process (InputStream in, String uri);

In fact, if your implementation does not require additional attributes
that need initialising, you do not even need to worry about
implementing init().

2. Program control flow

When a calling party (e.g. lempinen.neatseeker.core.NeatMaker)
requests an Indexer using the IndexerFactory.getIndexer static method,
a new Indexer is created according to the configuration parameters
(see section 4 below) and its init() method is called with the
Configuration object as an argument. Furthermore, the Factory sets the
Collector partner for the Indexer according to the settings in the
Configuration.

To start the actual indexing, the calling party calls the start()
method of the Indexer object. The default action in AbstractIndexer is
to simply continue by calling the start() method of the Collector, and
there should be no reason to override this.

The Collector then starts traversing either the filesystem or a
networked resource and calls back the process() function for each
resource it encounters.

3. Processing the data

This way, you can implement a simple indexer by specifying appropriate
actions for process(InputStream in, String uri).

The InputStream is your indexable data. The URI string is given by
the Collector as a unique identifier for the indexable resource. The
processing can the proceed as follows:

 - You should create a new Target in the beginning of process and use
   its setURI() method to set the URI to the string passed in by the
   Collector.

 - Use the repository lookahead method getNextID() to obtain a unique
   NeatSeeker identifier for the Target.

 - For every term you index, create an Entry that contains the term,
   the ID returned by getNextID() and the word position of the term
   within the resource.

 - Call the add(Entry) method in AbstractIndexer to add the term to
   the index.

 - At the end of processing a resource, update the index Statistics
   object with the incrementDocumentCount() and incrementWordCount()
   methods.

 - Update the word count of the Target with the setSize() method.

 - Put the Target in the repository with the repository.put() method.

Every call to process() should carry out the above tasks. For a
complete example on how this can be done, look at
lempinen.neatseeker.html.HTMLIndexer.

4. Adding your indexer to the runtime properties

As the Indexers are created by a Factory, the actual implementing
class is determined at runtime. You should tell NeatSeeker about your
new indexer by adding a few entries to the property file
(e.g. etc/NeatSeeker.properties).

For instance, if you have created an indexer for XML documents of some
sort, and the indexer class is com.abba.xml.XMLIndexer, you should add
the following entries to the property file:

    # Default indexer MIME
    neatseeker.indexer.mime		= text/xml

    # Fully qualified class names of different indexers
    neatseeker.indexer.text/xml	        = com.abba.xml.XMLIndexer

This should be enough to get you started. If you have problems, do not
hesitate to contact the author.