findex Java indexing tool

Introduction

Findex indexes HTML web pages, TXT text files, PDF files, Word 97+ DOC documents, PowerPoint PPT/PPS presentations and JPEG images for meta-data.  It does not currently index XLS files.  Findex does not have a search database editing facility.  Findex will not index Office documents if you run it using the Microsoft VM.

Findex supports Parser plug-ins to index new file types.  The supplied phdccRDF plug-in finds meta-data in RDF/XML files - this plug-in does run under the Microsoft VM.

Findex finds any meta-data information for field searches, including that contained in RDF/XML files.  A search database produced by Findex cannot be properly read by FindinSite-CD-Wizard because FindinSite-CD-Wizard does not support field searches.

Apart from fields, a search database produced by Findex will be functionally equivalent to one produced by FindinSite-CD-Wizard (with the same options selected).  However the search database physical files will not be identical.  Indeed, the search database files produced by different Java implementations (eg the Sun JVM and the Microsoft VM) will not be physically identical [because each implementation enumerates the contents of a hash table in a different order].

Findex is usually much slower than FindinSite-CD-Wizard on the same computer.  In addition, Findex will tend to use more memory than FindinSite-CD-Wizard.  The Microsoft VM seems to require less memory that the Sun JVM.  However the Sun JVM seems to run faster than the Microsoft VM for smaller sites.

Findex has been tested in Windows using the Microsoft and Sun JVMs, and in Sun Solaris 2.6 and Solaris 9 using the Sun JVM.  As described above, some aspects of Findex will not run under the Microsoft VM.

Executable

C:\Program Files\PHD\fisCD\Findex.jar

To use Findex on other platforms, you must first install the FindinSite-CD development kit on a Windows computer. Then copy Findex.jar from the installation directory to a suitable directory on the new platform.

You may need to add Findex.jar or its directory to the Java CLASSPATH environment variable definition if you are not using its full path when running it.

Usage

• Make a new search database
• Reindex an existing search database

Findex is supplied in a Java archive file called Findex.jar, with the main class com.phdcc.findex.Findex identified as the "Main-Class" in the JAR manifest.

Run Findex in the Java Virtual Machine on your computer, passing parameters to tell it what to do.  There should be no need to make any further changes to your existing Java CLASSPATH definition provided it already refers to the standard Java Runtime Environment classes.  For the Sun JVM you may need to use the -Xmx parameter to the java command to increase the maximum size of the memory allocation pool from 64MB if you run into Out of Memory errors.

Here are four different examples.  The first uses the Microsoft VM.  The last three use the Sun JVM in different ways.  In each case, the parameters are shown in green.

jview /cp:a Findex.jar com.phdcc.findex.Findex @index.properties
 
java -Xmx256M -jar Findex.jar -r MySite.his
 
java -cp Findex.jar;phdccRDF.jar com.phdcc.findex.Findex -q @index.properties
 
java -jar C:\Program Files\PHD\fisCD\Findex.jar -v

See the parser plug-ins documentation for details of how to run Findex with plug-ins.

Parameters

-v | -vv
or
[-q | -qq] ( @index_instructions_file | -r search_database )

If -q is specified then Findex runs in quiet mode, not writing any progress information to the "standard output" device, but still writing error information to the "standard error" device. If -qq is specified then Findex runs in silent mode, not writing any progress information to the "standard output" or "standard error" devices.

You must then specify whether you want to create a new search database (using @index_instructions_file) or reindex an existing search database (using -r search_database).  In fact, you can create a new search database to overwrite an existing one.

Reindexing a search database

Note that the Include and Exclude settings from an Index Instructions File are not saved in the search database system file, so any earlier inclusions or exclusions will not be used during a reindex. Similarly, plug-in options are not stored so these will be lost if you do a reindex.

If the search database was created by FindinSite-CD-Wizard and included options to scan file types not supported by Findex, then Findex will stop immediately with return value 31.

Index Instructions File

The Index Instructions file specifies the indexing options that Findex should use, such as the pathname of the search database to create.  The instructions file is in the format of a Java properties text file, where each line specifies a property and a value separated by an equals sign, with various escape sequences recognised - see below.  Property names are case sensitive.  See also the full example.

You must give a SaveAsPathname.  Depending on the ScanType, you must specify other parameters as follows:

• dir: ScanDirectory and ScanDirLevels
• file: ScanPathname
• url: ScanURL

The following table details all the available properties:

Include and Exclude files

The initial list of acceptable files is determined by the appropriate HTML_Files, HTML_Files, DOC_Files, XLS_Files, PPT_Files or TXT_Files settings. Then:

• If an Include file-spec set is given, then only files meeting one of the given file-specs will be indexed.
• If an Exclude file-spec set is given, then any files meeting one of the given file-specs will not be indexed.
• Note that the Includes are processed first and the Excludes afterwards, so an Exclude file-spec takes precedence.

An individual file-spec can include zero or more * or ? wildcard characters, where ? matches exactly one character, and * matches zero or more characters. For example file???.ht* would match:
    file001.htm, file101.html and file111.ht
but not
    file1001.htm

A list of file-specs can be given directly in the property, or indirectly in a file.

Direct file-specs

Direct file-specs are semi-colon separated, eg:
Include=iso*;*12*
Exclude=file???.ht*

This specifies two Include file-specs and one Exclude file-spec.

Indirect file-specs in a file

An indirect value consists of @ followed by a file name, where file-specs are specified one per line in plain text. The above direct example may be expressed indirectly as follows:
Include=@includes.txt
Exclude=@excludes.txt

where includes.txt contains:
iso*
*12*

and excludes.txt contains:
file???.ht*

If an indirect file cannot be opened, an error message is printed to the standard output unless -q or -qq specified.

Example Index Instructions File

The search database is built by following links from the URL http://www.phdcc.com/ and indexing HTML files that meet this file spec *.htm,*.html,*.asp and TXT files that meet this file spec *.txt.

Description=\ Seiten und W\u00F6rter SaveAsPathname=D:\\inetpub\\wwwroot\\fiscd\\srchdb ScanType=url ScanURL=http://www.phdcc.com/ HTML_Files=*.htm,*.html,*.asp ParseTXT=true

SaveAsPathname=D:/inetpub/wwwroot/fiscd/srchdb

Example Output

jview /cp:a Findex.jar com.phdcc.findex.Findex @index.properties
Findex 4.0, 29 July 2005.  Copyright (c) 2002-2011 PHD Computer Consultants Ltd
iso8859-3.htm
iso8859-15.htm
Windows874.htm
Windows1254.htm
iso8859-9.htm
Windows1258.htm
iso8859-1.htm
Windows1252.htm
Windows1256.htm
iso8859-6.htm
Windows1257.htm
iso8859-4.htm
Windows1250.htm
iso8859-5.htm
Windows1251.htm
Windows1253.htm
iso8859-7.htm
Windows1255.htm
iso8859-8.htm
iso8859-8i.htm
Post-processing...
No problems while scanning.

Pages scanned:			20  (208kB)
Words found:			1413
Total output database size:		98kB  (47%)
Time taken:			0 minutes, 2 seconds.

Error code 0

Index Instructions File Format

The stream is assumed to be using the ISO 8859-1 character encoding.

Every property occupies one line of the input stream. Each line is terminated by a line terminator (\n or \r or \r\n). Lines from the input stream are processed until end of file is reached on the input stream.

A line that contains only whitespace or whose first non-whitespace character is an ASCII # or ! is ignored (thus, # or ! indicate comment lines).

Every line other than a blank line or a comment line describes one property to be added to the table (except that if a line ends with \, then the following line, if it exists, is treated as a continuation line, as described below). The key consists of all the characters in the line starting with the first non-whitespace character and up to, but not including, the first ASCII =, :, or whitespace character. All of the key termination characters may be included in the key by preceding them with a \. Any whitespace after the key is skipped; if the first non-whitespace character after the key is = or :, then it is ignored and any whitespace characters after it are also skipped. All remaining characters on the line become part of the associated element string. Within the element string, the ASCII escape sequences \t, \n, \r, \\, \", \', \ (a backslash and a space), and \uxxxx are recognized and converted to single characters. Moreover, if the last character on the line is \, then the next line is treated as a continuation of the current line; the \ and line terminator are simply discarded, and any leading whitespace characters on the continuation line are also discarded and are not part of the element string.

As an example, each of the following three lines specifies the key "Truth" and the associated element value "Beauty":

Truth = Beauty
	Truth:Beauty
Truth			:Beauty

As another example, the following three lines specify a single property:

fruits	apple, banana, pear, \
	cantaloupe, watermelon, \
	kiwi, mango

The key is "fruits" and the associated element is:

"apple, banana, pear, cantaloupe, watermelon,kiwi, mango"

Note that a space appears before each \ so that a space will appear after each comma in the final result; the \, line terminator, and leading whitespace on the continuation line are merely discarded and are not replaced by one or more other characters.

As a third example, the line:

cheeses

specifies that the key is "cheeses" and the associated element is the empty string.

Return values

0	No errors or problems
-ve	No errors, but negative number indicates number of scan problems
1	Insufficient arguments
2	Invalid argument
4	Out of Memory
5	Unexpected exception
10	Error reading Index Instructions File
11	ScanDirectory invalid
12	SaveAsPathname not given in Index Instructions File
13	ScanType invalid
14	ScanDirectory property not specified
15	ScanDirLevels invalid
16	ScanPathname property not specified
17	ScanURL property not specified
18	Invalid true/false value for a property
19	AbstractWords invalid
20	Invalid parser plug-in
30	Error reading reindex parameters from existing Search database
31	Unable to reindex because an option is set that is not compatible with Findex, eg Parse PDF
100	General scan error
101	Internal error

In Windows, the Sun JVM java command seems to return 0 if it runs out of memory, and 3 if Ctrl+C is used to halt the program execution.  In Windows, the Microsoft VM jview command seems to return 3221225786 / -1073741510 / 0xC000013a / STATUS_CONTROL_C_EXIT if Ctrl+C is used to halt the program execution.  Out of memory might be 0xC0000017 / STATUS_NO_MEMORY.