findinsite-cd international character support

Introduction

Browsers and other viewers

Your first task is to ensure that users can read your files correctly. This is not a trivial task. Even though most web browsers can understand web pages written for different languages, they may not be display the information correctly on screen because the user does not have the relevant fonts or language support installed on their computer.

One option here is to use a different file format such as Adobe PDF or Microsoft Word. Files in these formats may render better on a user's screen than a web page. Adobe PDF also lets you control the page layout much more closely.

File formats

The starting point for representing characters correctly is to use the correct file format.

For HTML web pages, the key is to specify and use the most appropriate character set for the characters you are using. Most web page editors should let you specify the character set and encode characters in the character set correctly.

For other formats, such as Adobe PDF or Microsoft Word, the authoring tools may give you some scope to specify which fonts etc are used when saving a file.

FindinSite character support

• The FindinSite and Findex indexing programs support a wide range of characters.
• FindinSite-CD-Wizard also lets you view and edit a search database in your computer's default character format.
• The FindinSite runtimes display characters as best as possible for the user's browser and computer.
• The FindinSite runtimes are supplied fully internationalised for many languages as described in the Language support page.

FindinSite and Findex character support

HTML Web pages

A character set (charset) indicates how the contents of a web page should be interpreted as characters. Different languages will need different character sets if they contain different characters. The Unicode character set is a superset of most other character sets, but is not usually used to encode characters in web pages because it makes the pages larger.

TXT text files

FindinSite and Findex find all characters in single byte ANSI and double byte Unicode plain text files.

PDF Adobe files

Please see the PDF Scanning page for details of the character encodings that are recognised by the FindinSite and Findex indexers.

DOC, XLS and PPT Office documents

FindinSite and Findex extract all character information from the supported Microsoft Office documents.

HTML Character set basics

For web pages, the FindinSite and Findex indexers recognise 32 different character sets (and many more name variants), including multi-byte character sets. FindinSite-CD can display all characters if your computer and browser are set up correctly with the right fonts available. FindinSite-JS and FindinSite-MS use the UTF-8 character set, giving the best chance of displaying characters correctly.

As an example of character sets, Central European languages might need to use a capital R character with an acute accent above it, Ŕ. In the "Windows 1250" character set, byte code 192 (0xC0 in hexadecimal) represents this character. If the default character set were used instead, then this byte would be displayed as a capital A with a grave accent, ie À.

The default character set for web pages is called "ISO 8859-1". This is almost exactly the same as the standard Windows English character set called "Windows 1252".

Note that some character sets now include a Euro currency symbol (€) with byte code 128, ie 0x80 in hexadecimal (or 0x88 for the Windows-1251 charset). Older browsers may not be able to display this character.

Character normalisation

Internally, FindinSite and Findex use the Unicode character set. Each Unicode character uses 16 bits, ie a value from 0 to 65535, or U+0000 to U+FFFF to use the correct hexadecimal notation. The indexers translate characters from the page's character set into Unicode before processing them further. The search database uses a compact form of Unicode called UTF-8 that usually reduces the amount of storage space required.

As an example, the Euro currency symbol (€) is U+20AC, hexadecimal 0x20AC. The indexers translate byte code 128 in the "Windows 1252" character set into U+20AC. They also recognise other ways of specifying characters, using 'character references'. The Euro symbol may also be specified as a string €, as a decimal number € or as a hexadecimal number €.

Specifying character sets

You specify the character set for a web page using the following line, which must appear in the HEAD section of a web page. Replace ISO-8859-1 with an appropriate character set string. The list of character sets supported by the indexers is given below.
View the source of this page to see an example of this tag.

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">

Supported character sets

The indexers automatically detect pages that are encoded in the UTF-8, Unicode or UnicodeFEFF character sets when it starts to read a page. However the character set used for decoding may be overridden using the above META character set tag.

Email support if you would any other character sets supported.

Viewing characters in FindinSite-CD

To be more precise, your customers must have a computer set up as follows for satisfactory use of FindinSite-CD.

1. Install the appropriate fonts for the language you want to display.
2. If you run a version of Windows that can change locales, you need to change the default system locale to match the language.
3. The browser must be set up to use a suitable font for pages in the desired language.
4. The browser's Java fonts may need to be altered.

Internet Explorer

Windows Internet Explorer will display non-Western characters in its main browser window and the Microsoft Java VM will display and work with non-Western characters.

In Windows 2000 the default system locale must be set appropriately for this to work, eg set the default system locale to Japanese if you want to use FindinSite-CD to search for Japanese words. You must also install the Japanese Input Locale if you want to type in Japanese characters. In Windows XP, these steps do not seem to be necessary.

In Internet Explorer 5+, you can change the fonts that are used to display non-Western HTML. Select "Tools+Internet Options...". In the "General" tab press the "Fonts..." button. Select the character set type (eg Japanese) and then select a "Web page font" and a "Plain text font".

If you run FindinSite-CD in Internet Explorer in a computer that is not set up appropriately, FindinSite-CD will not display characters correctly.

Note: when running Internet Explorer in Windows 95, 98 and Me, you may well be able to view pages with non-Western characters by installing the appropriate language plug-in.

Sun Java VM

The Sun Java VM is used by most browsers, and can be used by Windows Internet Explorer. The Sun Java VM can display non-Western characters properly on Western computers.

VM 1.5 or later

To ensure that the right fonts are used for your language, simply open the Control Panel "Regional and Language Settings". In the "Regional Options" tab, select your language and country and press OK. (In some cases you may need to select "Supplemental language support" checkboxes in the "Language" tab, and appropriate Code pages in the "Advanced" tab.)

Restart the browser.

VM 1.4 and earlier

The Sun Java VM can display non-Western characters properly on Western computers, if the correct font properties are selected; you may be able to enter non-Western characters and search for them successfully.

If your computer is non-Western, then the Sun Java VM installation may have selected the correct font file already.  If not, then follow these instructions: First, find where the Java VM plug-in is installed.  On Windows computers, this might be in C:\Program files\Java\j2re1.4.2_01\.  Move to the lib\ sub-directory. The file font.properties determines which font the Java VM uses.  Save a copy of the current file, eg to font.properties.save.  The table on the right lists some alternative font properties files you could use.  For example, to use Japanese fonts, copy font.properties.ja to font.properties. Restart the browser.

Western font.properties Japanese font.properties.ja Korean font.properties.ko Simplified Chinese font.properties.zh Traditional Chinese font.properties.zh_TW

Scanning and viewing characters in FindinSite-CD-Wizard

However you will only be able to view characters correctly in FindinSite-CD-Wizard if Windows is set up appropriately for the characters that you are trying to display. Nonetheless, you can still edit "undisplayable" characters. So you can edit Japanese characters on a Western PC - if you are careful.

If a character cannot be displayed using the default system code page (character set), FindinSite-CD-Wizard does one of two things:

• In the Words list (and other places) FindinSite-CD-Wizard displays the character as a question mark (?). When you select a word in the words list, FindinSite-CD-Wizard also displays the word beneath the "Key" symbols. It uses the font that you have selected for printing to display the character. It is possible therefore that the character will be displayed correctly here, while not displayed correctly in the main words list.

  Windows 2000 and XP In this example, a Chinese character, U+5BFC 导, is listed as a word in the Words tab. FindinSite-CD-Wizard is being run in Windows 2000. Windows 2000 is not using Chinese as its system default locale so FindinSite-CD-Wizard cannot display this character properly in the words list. Therefore a ? is displayed instead in the main words list. However the chosen printer font, Simsun, can display this character. So when the character is selected in the words list, the character is displayed correctly under the "Key". In the western versions of Windows 95, Windows 98 and Windows Me you will never be able to view this character properly.

• For the Description and various entries in the Pages tab, FindinSite-CD-Wizard sees if any characters in the string cannot be displayed in the default code page. If there are any problems, FindinSite-CD-Wizard displays the entire string in UTF-8. This replaces any non-ASCII characters with two or three other characters. "UTF-8" is shown in the FindinSite-CD-Wizard status bar.

  You can edit the UTF-8 string - carefully. Suppose that the first character in the description is a Japanese character that cannot be edited normally. FindinSite-CD-Wizard uses its UTF-8 equivalent, ie three characters. If you delete the first of these characters, FindinSite-CD-Wizard displays the following message in the status bar below Could not decode the edit box string. If you go on to delete the other two characters, FindinSite-CD-Wizard will be able to decode the UTF-8 and so the status bar error message will go away.

  	Windows 2000 and XP(Japanese): Editing Japanese characters This example shows FindinSite-CD-Wizard successfully showing and editing Japanese characters. The selected title in the Pages tab is shown correctly both in the list box above and the edit box below. For this example, FindinSite-CD-Wizard was run in Windows 2000 with Japanese as the default system locale.
  	Windows 2000 and XP (Japanese): Editing Chinese characters In this example showing the Pages tab, FindinSite-CD-Wizard is again being run in Windows 2000 with Japanese as the default system locale. However, in this case, FindinSite-CD-Wizard is trying to display some Chinese characters. Some Chinese characters can be displayed in the Japanese default system locale, but not all. In the Pages list, the page title is shown with any unrecognised characters changed into question marks. The second, fourth and last characters are shown this way. However, in the Title box below, the page title is all shown in UTF-8 form. You can edit the title - in UTF-8 form - in this box.
  	Windows 2000 and XP (Chinese): Editing Chinese characters This example shows the same Chinese characters when FindinSite-CD-Wizard is run in Windows 2000 with Chinese as the default system locale. All the characters are displayed properly and can be edited easily.
  	Windows 98 (English): Editing Chinese characters This example shows the same Chinese characters when FindinSite-CD-Wizard is run in Western Windows 98. None of the characters can be displayed properly in the standard font, so all the non-space characters appear as question marks. The Title box again shows the page title in UTF-8 form. Even though exactly the same UTF-8 characters are output, Windows displays them differently in the default Western character set font.

Testing in Windows 2000 and XP

Windows 2000 and XP have features that let you test your non-Western FindinSite-CD implementation fully. The first step is to install the locales of interest. From the Control Panel, select the "Regional Settings" applet. In the "General" tab, check the "Language settings for the system". Make sure that all the locales of interest are installed. If necessary, press the "Advanced" button to see what Code Pages are installed.

In FindinSite-CD-Wizard, you should now be able to File+Print all the characters in your search database. You may well need to change the print font, in the "File+Print options..." dialog box. The following fonts may be useful: "Microsoft Sans Serif", "Ludica sans Unicode" or "Arial Unicode MS" might display most characters, "MS Mincho" displays Japanese and "Simsun" displays Chinese. You should also now be able to see the selected Word displayed correctly under the "Key".

FindinSite-CD-Wizard will not yet be able to display and edit non-Western characters easily (although you can view and edit in UTF-8). To make this work you will have to set the default system locale appropriately. In the "Regional Settings" "General" tab, press the "Set default". Choose the desired locale. You will probably have to reboot to bring this change into effect.
Note that changing "your locale" in the "Regional Settings" "General" tab is usually not sufficient to make FindinSite-CD-Wizard work well.

As a final step, you will need to add the appropriate input locales. In the "Regional Settings" box, select the "Input Locales" tab. Add any locales that you need. Each input locale has different ways of entering characters, called Input Method Editors (IME).

To run FindinSite-CD in a browser in Windows 2000 and XP, see the instructions above.

Unicode implementation details

Unicode and UTF-8

Canonicalisation

The indexers convert all half-width Katakana and Hangul characters into their standard width character codes. Many other useful character code translations are also done.

The FindinSite runtimes also perform these same translations, so that if you enter a Japanese full width A, the search process correctly matches the latin A.

Possible improvement: Translate traditional Chinese character into their simplified Chinese equivalent when doing searches.

Word Splitting

Each non-latin character (ie non-Western and non-Arabic) forms a separate word, as described in the next section.

Non-western characters

This approach is used because most non-Western languages do not use white space characters to indicate breaks between words.

A character is defined as being Western if its Unicode code is less than or equal to U+02A8, or if it is in the Unicode range U+1E00 to U+1EF9 inclusive.

HTML Tag and Tag Attribute names

Although web page names and target frame names can include any characters, it is recommended that they be in Western characters.

Lower Case

Stop words

FindinSite-CD Word highlighting

For Microsoft Internet Explorer 4 or later, the browser decodes the character sets before word highlighting. This makes it possible to do word highlighting whatever the character set, ie word highlighting works in IE4+.

Word highlighting in Netscape Navigator 4 or similar only works if the character set is "iso-8859-1" or "Windows-1252". For other character sets, word highlighting is not attempted because FindinSite-CD does not decode the charset at runtime.