Release 2.3
A58165-01
 Library |
 Product |
 Contents |
 Index |
| Oracle8 ConText Cartridge Administrator's Guide Release 2.3 A58165-01 |
|
This chapter introduces the concepts necessary for understanding the objects in the ConText data dictionary.
The following topics are discussed in this chapter:
This section provides conceptual, as well as reference, information about policies, which are stored in the ConText data dictionary:
A policy is a logical grouping of six indexing preferences (one preference for each of the supported categories), assigned to a column in the database. A policy specifies the options used by ConText to create the index for the text in the column. It is also used to generate linguistic information for use in ConText applications.
Policies can be created by any ConText user with the CTXAPP role. Policies are stored in the ConText data dictionary. In addition to the preferences for a policy, users specify a name for the policy and the text column for the policy, and a number of other policy attributes.
The policies created by a user must be unique for the user. As such, the same policy for a user cannot be assigned to more than one column.
A column policy is a policy that has a text column assigned to it. Only column policies can be used to create ConText indexes or generate linguistic output.
Multiple policies, as long as they are unique for the user, can be assigned to a column. As a result, a column can have more than one index.
When a query is performed, you can specify a policy name to indicate the index that is used to process the query.
This feature is particularly useful if you have English-language documents for which you want to enable both text and theme queries. To enable text and theme queries, you must create both a text indexing policy and a theme indexing policy on the column containing the documents and create a ConText index for each policy.
A template policy is a policy that does not have a text column. Template policies are stored in the ConText data dictionary and are owned by the user who created them.
A template policy can be used by the policy owner as a source policy when creating new column or template policies. When a template policy is used as a source policy in a new policy, all of the preferences for the template policy are copied to the new policy. Any preference from the template policy can be overridden by explicitly naming a preference (for the same category) during the creation of the new policy.
A text indexing policy is any policy created with a Lexer preference that uses the BASIC LEXER Tile or one of the Tiles provided for the pictorial languages supported by ConText.
Once a text index is created for the policy, any text requests, including text queries, on the policy will result in the text index being accessed.
|
See Also:
For an example of creating a text indexing policy, see "Creating a Column Policy" in Chapter 9, "Setting Up and Managing Text". For more information about text indexes, see "Text Indexes" in Chapter 6, "Text Concepts". For more information about text queries, see Oracle8 ConText Cartridge Application Developer's Guide. |
By specifying the THEME LEXER Tile in the Lexer preference used in a column policy, you designate the policy as a theme indexing policy.
In addition, stoplists are not used by the theme lexer, so a NULL Stoplist preference can be specified for the policy.
Once a theme index is created for a theme indexing policy, any text requests, including queries, on the policy will result in the theme index being accessed.
|
See Also:
For an example of creating a theme indexing policy, see "Creating a Theme Indexing Policy" in Chapter 9, "Setting Up and Managing Text". For more information about theme indexes, see "Theme Indexes" in Chapter 6, "Text Concepts". For more information about theme queries, see Oracle8 ConText Cartridge Application Developer's Guide. |
Consider a table with two text columns: one holds Microsoft Word documents and the other holds (plain text) comments for the documents. The table structure is:
To create a text index for both the comment and doc columns in doc_and_comment, a policy must be defined for each column. The following example illustrates two policies named i_doc and i_comments that could be created:
To create a theme index for the doc column, a theme indexing policy must be defined. The following example illustrates a policy named i_theme that could be created for the table:
To define a policy, a user specifies a name for the policy and a number of optional attributes.
Because a policy is owned by the user who creates it, the policy name must be unique for a user; however, different users can have policies with the same name.
The following policy attributes are optional:
specifies the column in a table to which a policy is assigned. It is the column used to store text in the table.
species a description of the policy.
specifies the primary key column or columns (up to sixteen) for the table. This attribute is required if the policy is being assigned to a column.
specifies the column storing the unique identifier for the text column in a master-detail table. A master-detail table does not store a document as a single row, but rather breaks the document (identified by the textkey) into sections and stores each section in a separate row in the table. The collection of rows with the same textkey represents the whole document.
|
Note: This attribute is used only for policies that include a preference for the MASTER DETAIL Tile. |
specifies an existing template policy that you want to use as the basis for a new policy. When you specify a source policy in a policy, all of the preferences for the template (source) policy are copied into the new policy. The preferences from the source policy can be overwritten by explicitly specifying a preference for the category.
To define a policy, the user specifies a preference for each of the six supported categories. ConText does not require the user to specify a preference for the seventh category, Compressor, because data compression is not currently supported.
A preference can be used in more than one policy; however, two preferences from the same category cannot be used in the same policy.
In a policy, if a user does not specify a preference for one of the preference categories, ConText uses the default preference for the category.
The above figure illustrates how the default preferences and user-specified preferences work together to create a complete policy.
ConText provides the following template policies (listed in alphabetical order):
This template policy uses all of the default preferences. It can be used to create a policy with the following characteristics:
|
Note: DEFAULT_POLICY is the default for source_policy in both CTX_DDL.CREATE_POLICY and CTX_DDL.CREATE_TEMPLATE_POLICY. |
This template policy uses the AUTOB predefined preference and all the remaining default preferences. It can be used to create a column policy for a text column that contains documents in mixed formats.
This template uses the following predefined preferences and can be used to create a column policy which enables basic section searching for a text column containing HTML documents:
This template policy uses all the default preferences. It can be used to create a policy for indexing basic text stored in a text column.
This template policy uses the NO_STOPLIST predefined preference and all the remaining default preferences. It can be used to create a policy that does not use a stoplist during indexing.
This template policy uses the DEFAULT_STOPLIST predefined preference and all the remaining default preferences. It can be used to create a policy that uses the supplied English stoplist during indexing.
This template policy use the MD_TEXT predefined preference and all the remaining default preferences. It can be used to create a policy for indexing text stored in the detail column in a master-detail table.
This template policy uses the MD_BINARY predefined preference and all the remaining default preferences. It can be used to create a policy for indexing text stored in the detail column in a master-detail table.
This template policy uses the WW6B predefined preference and all the remaining default preferences. It can be used to create a policy for indexing text in Microsoft Word for Windows 6 format.
This section provides conceptual, as well as reference, information for indexing preferences, which are stored in the ConText data dictionary:
Indexing preferences specify the options that ConText uses to create ConText indexes. Each preference represents one (and only one) indexing option.
A preference consists of a ConText Tile and one or more attributes (and their corresponding values) for the Tile.
In addition, each preference is grouped into one of six types or categories, which determine the indexing operation that the preference controls. While a category is not explicitly assigned to a preference, it is implied through the association of the Tile with the preference.
When creating a policy, six preferences are specified for the policy, one for each of the six categories.
A ConText user with the CTXAPP role can create their own preferences by setting the required attributes for one of the Tiles provided by ConText, then calling CTX_DDL.CREATE_PREFERENCE and specifying the name of the Tile.
ConText provides a number of predefined preferences (owned by CTXSYS) for each category. These predefined preferences can be used by any ConText user with the CTXAPP role to create policies without having to first create preferences.
Tiles are the objects in the ConText data dictionary that provide ConText servers with information about how text is managed in the system, as well as indexing instructions. Each Tile specifies a distinct indexing option within the ConText framework.
A Tile is the main component of a preference. When you define a preference, you specify a Tile and attributes for the Tile, as well as a value for each attribute.
Each Tile may have none, one, or many attributes that are specified to define a preference. The attributes identify which indexing options are active for the Tile in a preference.
Each Tile attribute has a value (either a number or a string) that you assign when you specify attributes in a preference.
The indexing options that must be specified for ConText are divided into seven functional categories or classes.
Each category contains one or more Tiles for which you specify attributes when creating preferences. The Tiles in the categories essentially provide answers to the questions necessary for ConText to generate an index for a text column:
ConText provides the following predefined Data Store preferences:
This preference calls the DIRECT Tile, which is used to indicate that text is stored directly in the text column of a text table.
This preference calls the OSFILE Tile, which is used to indicate that text is stored as files in a file system,
DEFAULT_OSFILE uses the path attribute and a hardcoded set of dummy directory paths to indicate the directories in which the text files are located.
The hard-coded paths, delimited by colons are: /oracle/data, /oracle/data2, /oracle/data3.
This preference calls the URL Tile which is used to indicate that text is stored as URLs.
DEFAULT_URL uses all of the attribute defaults for the URL Tile:
This preference calls the MASTER DETAIL Tile which is used to indicate text is stored in a master detail table.
MD_BINARY uses the binary attribute and a value of YES to indicate that the text in the table is stored in binary format (newline characters do not indicate end of line).
This preference calls the MASTER DETAIL Tile which is used to indicate text is stored in a master detail table.
MD_TEXT uses the binary attribute and a value of NO to indicate that the text in the table is stored in plain text format (newline characters indicate end of line).
ConText provides the following predefined Filter preferences:
This preference calls the BLASTER FILTER Tile which specifies an internal filter used to extract text from formatted documents in a text column.
AUTOB uses the format attribute and a value of 997 to indicate that ConText uses the autorecognize filter to extract text. It can be used to filter text in a column that contains the following document formats:
This preference is identical to the HTML_FILTER predefined preference, except the keep_tag attribute is set with the following values to support basic section searching in HTML documents:
This preference calls the FILTER NOP Tile which indicates that the text column in a text table contains plain, unformatted (ASCII) text and does not require filtering for indexing and highlighting.
This preference calls the HTML FILTER Tile and can be used to filter documents in a column that contains only HTML-formatted documents.
This preference calls the BLASTER FILTER Tile and specifies a value of 11 for the format attribute to indicate ConText uses the Word for Windows 6 filter to extract text. It can be used in a column that contains only Word for Windows 6-formatted documents.
ConText provides the following predefined Lexer preferences:
This preference is identical to DEFAULT_LEXER, except the startjoins and endjoins attributes for the BASIC LEXER Tile are set with '</' and '>' respectively to support basic section searching in HTML documents.
This preference calls the BASIC LEXER Tile, which indicates the lexer settings used to identify word and sentence boundaries for text indexing and text queries.
DEFAULT_LEXER uses the following Tile attributes and values to indicate the lexer settings:
This preference calls the KOREAN LEXER Tile and can be used for parsing Korean text. It has no attributes.
This preference calls the THEME LEXER Tile, which indicates the preference can be used in a column policy to create theme indexes for a column.
The THEME_LEXER preference does not set any attributes because the THEME LEXER preference doesn't have any attributes.
This preference call the CHINESE V-GRAM LEXER Tile, which indicates the preferences can be used for parsing Chinese text.
The 1 or 2 indicates that the preference uses either method 1 or 2 for identifying tokens in Chinese text (hanzi_indexing attribute).
This preference call the JAPANESE V-GRAM LEXER Tile which indicates the preferences can be used for parsing Japanese text.
The 1 or 2 indicates that the preference uses either method 1 or 2 for identifying tokens in Japanese text (kanji_indexing attribute).
ConText supplies a single predefined Engine preference, DEFAULT_INDEX.
This preference calls the GENERIC ENGINE Tile which is used to specify the amount of memory reserved for indexing.
DEFAULT_INDEX uses the index_memory attribute and specifies the amount of memory allocated for indexing: 12582912 bytes.
ConText provides the following predefined Wordlist preferences:
This preference is identical to the NO_SOUNDEX preference, except the section_group attribute has a value of 'BASIC_HTML_SECTION', which is a predefined section group provided by ConText for basic section searching of HTML text.
This preference specifies a value of 0 for the soundex_at_index attribute to indicate that ConText does not generate Soundex word mappings during text indexing.
This preference specifies a value of 1 for the soundex_at_index attribute to indicate that ConText generates Soundex word mappings during text indexing.
This preference specifies a value 3 for the fuzzy_match attribute to ensure fuzzy matching is not enabled for Korean.
This preference specifies a value 4 for the fuzzy_match attribute to ensure fuzzy matching is not enabled for Chinese.
This preference specifies a value 2 for the fuzzy_match attribute to enable fuzzy matching for Japanese.
ConText provides the following predefined Stoplist preferences for creating text indexes:
|
See Also:
For a complete list of the stop words for each of the predefined Stoplist preferences, see Appendix A, "Supplied Stoplists". |
This preference defines a list of English terms treated as stop words during indexing.
|
Note: The stop words in DEFAULT_STOPLIST are in all-uppercase. As a result, DEFAULT_STOPLIST should be used only when creating case-insensitive text indexes. If you are creating case-sensitive text indexes and want to use a stoplist during indexing, first create a Stoplist preference that includes all forms (i.e. uppercase, initial uppercase, lowercase) of the words to be processed as stop words. For more information about case-sensitivity in text indexes, see "What's in a Text Index?" in Chapter 6, "Text Concepts" |
This preference specifies that no list of stop words is used during text indexing. All words that ConText encounters are stored in the text index.
This preference defines a list of French terms treated as stop words during indexing.
This preference defines a list of German terms treated as stop words during indexing.
This preference defines a list of Italian terms treated as stop words during indexing.
This preference defines a list of Spanish terms treated as stop words during indexing.
|
Note: The stop words in FRENCH_STOPLIST, GERMAN_STOPLIST, ITALIAN_STOPLIST, and SPANISH_STOPLIST are in all-lowercase. As a result, these preferences can be used in case-sensitive and case-insensitive text indexes; however, if they are used in case-sensitive indexes, stop words that appear in all-uppercase or initial uppercase in your text will not be processed as stop words. To ensure all stop words are processed correctly in a case-sensitive text index, create a Stoplist preference that includes all forms (i.e. uppercase, initial uppercase, lowercase) of the words to be processed as stop words. For more information about case-sensitivity in text indexes, see "What's in a Text Index?" in Chapter 6, "Text Concepts" |
Data Store Tiles are used to create preferences which specify how text (data) is stored in the database. ConText supports the following methods of storing text in the database:
|
See Also:
For more information about text storage, see "Text Storage" in Chapter 6, "Text Concepts". |
ConText provides the following Data Store Tiles:
The DIRECT Tile is used for text stored directly in the database. It has no attributes.
The MASTER DETAIL Tile is used for text stored directly in the database in master-detail tables, with the textkey column located in the detail table. The column policy is assigned to this column.
MASTER DETAIL has the following attribute(s):
The binary attribute specifies whether the text in a master detail table is in plain text format (0) or binary format (1).
Text in plain text format uses newline characters at the end of each line to indicate the end of the line. In contrast, binary format does not use newline characters to indicate the end of the line.
The MASTER DETAIL NEW Tile is used for text stored directly in the database in master-detail tables, with the textkey column located in the master table. The column policy is assigned to this column and all detail information is stored in the Data Store preference, rather than the column policy.
MASTER DETAIL NEW has the following attribute(s):
The binary attribute specifies whether the text in a master detail table is in plain text format (0) or binary format (1).
The detail_table attribute specifies the name of the detail table in the master-detail relationship.
The detail_key attribute specifies the name of the foreign key column in the detail table.
The detail_lineno attribute specifies the name of the column in the detail table that identifies rows in the table.
The detail_text attribute specifies the name of the text column in the detail table.
The OSFILE Tile is used for text stored in files accessed through the local file system.
OSFILE has the following attribute(s):
The path attribute specifies the location of text files that are stored externally in a file system.
Multiple paths can be specified for path, with each path separated by a colon (:). File names are stored in the text column in the text table. If path is not used to specify a path for external files, ConText requires the path to be included in the file names stored in the text column.
The URL Tile is used for text stored:
URL has the following attribute(s):
The timeout attribute specifies the length of time, in seconds, that a network operation such as 'connect' or 'read' waits before timing out and returning a timeout error to the application. The valid range for timeout is 0 to 3600 and the default is 30.
maxthreadThe maxthreads attribute specifies the maximum number of threads that can be running at the same time. The valid range for maxthreads is 1 to 1024 and the default is 8.
maxurlsThe maxurls attribute specifies the maximum number of rows that the internal buffer can hold for HTML documents (rows) retrieved from the text table. The valid range for maxurls is 1 to 4294967295 and the default is 256.
The urlsize attribute specifies the maximum length, in bytes, that the URL data store supports for URLs stored in the database. If a URL is over the maximum length, an error is returned. The valid range for urlsize is 32 to 65535 and the default is 256.
maxdocsizeThe maxdocsize attribute specifies the maximum size, in bytes, that the URL data store supports for accessing HTML documents whose URLs are stored in the database. The valid range for maxdocsize is 1 to 4294967295 and the default is 200000 (2 Mb).
The http_proxy attribute specifies the fully-qualified name of the host machine that serves as the HTTP proxy (gateway) for the machine on which ConText is installed. This attribute must be set if the machine is in an intranet that requires authentication through a proxy server to access Web files located outside the firewall.
The ftp_proxy attribute specifies the fully-qualified name of the host machine that serves as the FTP proxy (gateway) for the machine on which ConText is installed. This attribute must be set if the machine is in an intranet that requires authentication through a proxy server to access Web files located outside the firewall.
The no_proxy attribute specifies a string of domains (up to sixteen, separate by commas) which are found in most, if not all, of the machines in your intranet. When one of the domains is encountered in a host name, no request is sent to the machine(s) specified for ftp_proxy and http_proxy. Instead, the request is processed directly by the host machine identified in the URL.
For example, if the string 'us.oracle.com, uk.oracle.com' is entered for no_proxy, any URL requests to machines that contain either of these domains in their host names are not processed by your proxy server(s).
The following example creates a preference named doc_ref for the OSFILE Tile:
begin ctx_ddl.set_attribute ('PATH', '/private/mydocs'); ctx_ddl.create_preference ('DOC_PREF', 'Path my for my documents' 'OSFILE'); end;
Filter Tiles are used to create preferences which determine how text is filtered for indexing and highlighting. Filters allow word processor and formatted documents, as well as ASCII and HTML text documents, to be indexed and highlighted by ConText.
For formatted documents, ConText stores documents in their native format and uses filters to build temporary ASCII versions of the documents. ConText indexes the temporary ASCII text of the formatted document. ConText also uses the ASCII version to highlight query terms.
ConText provides internal filters for processing many of the popular document formats, including Microsoft Word, WordPerfect, and AmiPro.
In addition, ConText allows users to specify external filters for filtering documents in formats not supported by the internal filters provided with ConText.
External filters can also be used to perform operations, such as cleaning up or converting text, before the text is filtered for indexing and highlighting.
|
See Also::
For examples of creating Filter preferences, see "Creating Filter Preferences" in Chapter 9, "Setting Up and Managing Text". For more information about internal and external filters, see "Text Filtering" in Chapter 6, "Text Concepts". |
ConText provides the following Filter Tiles:
The BLASTER FILTER Tile is used to specify that the internal filters are used to filter documents. It can also be used to specify that multiple external filters are used to filter documents in a mixed-format column.
BLASTER FILTER has the following attribute(s):
The format attribute specifies the internal filter used for filtering text stored in a text column.
The executable attribute specifies the external filters that are used to filter text stored in a mixed-format text column. It has three values that must be specified:
|
See Also:
For a list of the format IDs supported by the executable attribute, see "Supported External Filter Formats for Mixed-Format Columns" in Chapter 6, "Text Concepts". |
The FILTER NOP Tile is used to specify that plain text is stored in the text column and no filtering needs to be performed. It has no attributes.
The HTML FILTER Tile is used to specify that the internal HTML filter is used to filter plain text that contains HTML tags.
HTML_FILTER has the following attribute(s):
The code_conversion attribute specifies whether code conversion is enabled for documents which contain Japanese ASCII text with HTML tags.
Code conversion is required for Japanese HTML documents if the documents use more than one of the three character sets supported for HTML text in Japanese. If code conversion is enabled, all Japanese HTML documents are converted to a single, common character set before indexing.
The default for code_conversion is 0 (disabled).
keep_tagThe keep_tag attribute takes two values: the HTML tag to retain during indexing and a sequence number that uniquely identifies the tag.
The following rules apply to keep_tag:
For example, keep_tag is set to BODY and the following string occurs in a document:
<HTML><BODY BGCOLOR=#ffffff>hello</BODY></HTML>
ConText translates the string to:
<BODY>hello</BODY>
This string is passed to the HTML filter, which ignores the HTML tags, then to the lexer, which indexes the token hello as belonging to the BODY section.
The USER FILTER Tile is used to specify an external filter for filtering documents in a column.
USER FILTER has the following attribute(s):
The command attribute specifies the executable for the single external filter used to filter all text stored in a column. If more than one document format is stored in the column, the external filter specified for command must recognize and handle all such formats, otherwise the BLASTER FILTER Tile and the executable attribute should be used instead of the USER FILTER Tile.
The following section provides two Filter preference examples:
The following example creates a preference named word6 for the BLASTER FILTER Tile:
begin ctx_ddl.set_attribute ('FORMAT', '11'); ctx_ddl.create_preference ('WORD6', 'Microsoft Word docs', 'BLASTER FILTER'); end;
The following example creates a preference named sect_filt_pref for the HTML FILTER Tile:
begin ctx_ddl.set_attribute('KEEP_TAG', 'TITLE', 1); ctx_ddl.set_attribute('KEEP_TAG', 'HEAD', 1); ctx_ddl.set_attribute('KEEP_TAG', 'BODY', 1); ctx_ddl.set_attribute('KEEP_TAG', 'H1', 1); ctx_ddl.create_preference('sect_filt_pref','sect search filt','HTML FILTER'); end;
In this example, the <TITLE>, </TITLE>, <HEAD>, </HEAD>, <BODY>, </BODY>, <H1>, and </H1> HTML tags are retained by the HTML filter during filtering, provided the startjoins and endjoins attributes for the BASIC LEXER Tile are set appropriately.
The Tiles in the Lexer category are used to create preferences which specify the lexer used to perform indexing.
A text lexer parses text and identifies tokens for indexing. English and other single-byte languages, including most European languages, can use the same lexer because tokens (words) in those languages are delimited by blank spaces and standard punctuation (commas, periods, question marks, etc.).
Japanese, Chinese, and many other Asian languages are pictorial (multi-byte) languages that cannot be tokenized in the same manner as single-byte languages.
ConText includes a single Lexer Tile, BASIC LEXER Tile, for all of the single-byte languages, such as English (7-bit character set) and the other European languages (8-bit character sets), supported by ConText. The basic lexer also works with languages such as Greek, which have different alphabets, but still utilize blank spaces to delimit words.
ConText includes three separate Lexer Tiles for processing Japanese, Chinese, and Korean text.
The CHINESE V-GRAM LEXER Tile and JAPANESE V-GRAM LEXER Tile do not rely on finding token boundaries within text; instead, they uses a list of terms to match and index patterns of characters at user-specified, variable points of length.
The Japanese and Chinese lexers also work with languages that use a 7-bit character set, such as English. As a result, ConText supports indexing and querying Japanese and Chinese text that also contains English text.
The Korean lexer, KOREAN LEXER Tile, works similarly to the Japanese and Chinese lexers by finding character patterns in the text and matching the patterns to a dictionary of terms. However, due to the significant morphological transformations that Korean verbs undergo, the Korean lexer only indexes nouns and noun phrases.
The BASIC LEXER Tile supports all NLS-compliant character sets, including the AL24UTFFSS (UTF-8) character set. UTF-8 is a character set that recognizes the characters from most single-byte and multi-byte character sets.
Users with multi-lingual environments, such as multi-national companies, can specify UTF-8 for a database and use the database to store documents that use any one of the character sets supported by UTF-8. ConText supports indexing all documents stored in a UTF-8 database and queries to the database from clients running any of the UTF-8 supported character sets.
The BASIC LEXER Tile currently supports the UTF-8 character set only for space-delimited, single-byte languages, which includes English and other Western European languages.
The BASIC LEXER Tile does not support UTF-8 for the multi-byte languages, nor do the Japanese, Chinese, and Korean lexers currently support UTF-8.
The BASIC LEXER Tile does not require any setup to enable it to handle UTF-8 or other NLS-compliant character sets; however, the NLS_LANG environment variable must be set to the appropriate language/territory/character set. In addition, the ORA_NLS32 and ORA_NLS environment variables must be set to the directories containing the appropriate NLS data.
The lexer has the following limitations when UTF-8 is the character set specified for the database:
For German-language text, the BASIC LEXER Tile provides an attribute for enabling composite word indexing. With composite word indexing, tokens that are compound words (specifically, nouns in German text) are divided into their constituent (root) nouns, including inflected forms of the roots, and the roots are stored in the ConText index along with the entry for the compound word.
For example, if the word Hauptbahnhof is encountered in a document during composite word indexing, the following entries are created in the index: HAUPTBAHNHOF, HAUPT, BAHN, BAHNEN, HOF.
Composite word indexing supports both single-byte and multi-byte character sets, specifically WE8ISO8859P9 (extended, single-byte) and AL24UTFFSS (multi-byte).
Composite indexes have the following limitations:
Composite word indexing enables text queries to return all documents that contain either the query term itself or the query term as a root of a compound word; however, queries for phrases that contain one or more compound words return only the documents that contain the exact phrase.
|
Note:
For more information about composite word queries, see Oracle8 ConText Cartridge Application Developer's Guide. |
For English-language text, a separate lexer, THEME LEXER Tile, is provided for creating theme indexes. This lexer breaks text into tokens; however, the tokens are not stored in the theme index. The tokens are passed to the ConText linguistic core where they are analyzed within the context of the sentences and paragraphs in which they appeared to determine whether they are content-bearing words. The linguistic core then generates themes, which are stored in the theme index.
The themes generated by ConText are based on, but are not identical to, the content-bearing tokens in the text.
|
See Also:
For more information about the theme lexer and theme indexing, see "Theme Indexes" in Chapter 6, "Text Concepts". |
ConText provides the following Lexer Tiles for creating preferences for indexing:
The BASIC LEXER Tile is used to identify tokens for creating text indexes for English and all other supported single-byte languages. It is also used to enable base-letter conversion for single-byte languages that have extended character sets.
BASIC LEXER has the following attribute(s):
base_letter specifies whether characters that have diacritical marks (umlats, cedillas, acute accents, etc.) are converted to their base form before being stored in the text index. The default is 0 (base-letter conversion disabled).
continuation specifies the characters that indicate a word continues on the next line and should be indexed as a single token. The most common continuation characters are a hyphen '-' and a backslash '\'.
numgroup specifies the characters that, when they appear in a string of digits, indicate that the digits are groupings within a larger single unit.
For example, comma ',' or period '.' may be defined as numgroup characters because they often indicate a grouping of thousands when they appear in a string of digits.
numjoin specifies the characters that, when they appear in a string of digits, cause ConText to index the string of digits as a single unit or word.
For example, period '.' or comma ',' may be defined as numjoin characters because they often serve as decimal points when they appear in a string of digits.
printjoinsprintjoins specifies the non-alphanumeric characters that, when they appear anywhere in a word (beginning, middle, or end), are processed by ConText as alphanumeric and included with the token in the text index. This includes printjoins that occur consecutively.
For example, if the hyphen '-' and underscore '_' characters are defined as printjoins, terms such as pseudo-intellectual and _file_ are stored in the text index as pseudo-intellectual and _file_.
punctuationspunctuations specifies the non-alphanumeric characters that, when they appear at the end of a word, indicate the end of a sentence or a grouping within a sentence.
Characters that are defined as punctuations are removed from a token before text indexing; however, if a punctuations character is also defined as a printjoins character, the character is only removed if it is the last character in the token and it is immediately preceded by the same character.
For example, if the period (.) is defined as both a printjoins and a punctuations character, the following transformations take place during indexing and querying as well:
| Token | Indexed Token |
|---|---|
|
.doc |
.doc |
|
dog.doc |
dog.doc |
|
dog..doc |
dog..doc |
|
dog. |
dog |
|
dog... |
dog.. |
skipjoins specifies the non-alphanumeric characters that, when they appear within a word, identify the word as a single token; however, the characters are not stored with the token in the text index.
For example, if the hyphen character '-' is defined as a skipjoin, the word pseudo-intellectual is stored in the text index as pseudointellectual.
startjoins/endjoinsstartjoins specifies the characters that, when encountered as the first character in a token, explicitly identify the start of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the ConText index entry for the token. In addition, the first startjoins character in a string of startjoins characters implicitly end the previous token.
endjoins specifies the characters that, when encountered as the last character in a token, explicitly identify the end of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the ConText index entry for the token.
The following rules apply to both startjoins and endjoins:
|
Note: Defining startjoins and endjoins characters is particularly useful for creating document sections that enable section searching in a column. For examples of creating sections and section groups, see "Managing Document Sections" in Chapter 9, "Setting Up and Managing Text". For more information about sections, see "Document Sections" in Chapter 6, "Text Concepts". For more information about section searching, see Oracle8 ConText Cartridge Application Developer's Guide. |
The mixed_case attribute specifies whether the lexer converts the tokens in text index entries to all uppercase or stores the tokens exactly as they appear in the text. The default is 0 (tokens converted to all uppercase).
The composite attribute specifies whether composite word indexing is enabled. For the current release, composite indexing is supported for German-language text only. The default is 0 (no composite word indexing).
The CHINESE V-GRAM LEXER Tile is used for identifying tokens for creating text indexes for Chinese text.
CHINESE V-GRAM LEXER has the following attribute(s):
The hanzi_indexing attribute specifies the number of characters used for pattern matching while indexing.
A value of 1 for hanzi_indexing indicates that the Chinese lexer examines each character individually to determine token boundaries.
A value of 2 for hanzi_indexing indicates that the lexer examines characters in pairs to determine token boundaries. Pattern matching using pairs is generally faster than matching individual characters, resulting in faster index creation.
The default is 2.
The JAPANESE V-GRAM LEXER Tile is used for identifying tokens for creating text indexes for Japanese text.
JAPANESE V-GRAM LEXER has the following attribute(s):
The kanji_indexing attribute specifies the number of characters used for pattern matching while indexing.
A value of 1 for kanji_indexing indicates that the Japanese lexer examines each character individually to determine token boundaries.
A value of 2 for kanji_indexing indicates that the lexer examines pairs of characters to determine token boundaries. Pattern matching using pairs is generally faster than matching individual characters, resulting in faster index creation.
The default is 2.
The KOREAN LEXER Tile is used for identifying tokens for creating text indexes for Korean text. It has no attributes.
The THEME LEXER Tile is used to create theme indexes for English-language text. It has no attributes.
The following section provides two Lexer preference examples that both use the BASIC LEXER Tile.
The following example creates a preference named doc_link:
begin ctx_ddl.set_attribute ('PRINTJOINS', '.-@&$#/'); ctx_ddl.create_preference ('DOC_LINK', 'numerous joins', 'BASIC LEXER' ); end;
In this example, the '.', '-', '@', '&', '$', '#', and '/' characters are all defined as printjoins characters.
Characters such as the dollar sign '$' and number sign '#' are useful if you want to index tokens that may contain these characters, such as sums of money and numbers.
The following example creates a preference named section_pref:
exec ctx_ddl.set_attribute(`startjoins','</'); exec ctx_ddl.set_attribute(`endjoins','>'); exec ctx_ddl.set_attribute(`printjoins','_@-&$#.'); ... exec ctx_ddl.create_preference(`sect_lex_pref','basic lexing + sections','BASIC LEXER');
In this example, the characters `<` and '/' are defined as startjoins characters. The character `>' is defined as an endjoins character.
The open and closed angle brackets '< >' and the forward slash '/' are useful for identifying HTML tags for document sections.
|
See Also:
For more information about sections, see "Document Sections" in Chapter 6, "Text Concepts" |
Engine Tiles are used to create preferences which specify how ConText indexes are created by the ConText engine and where in the database the indexes are stored.
The engine is the ConText component that creates a ConText index for a text column. A ConText index is required before text in a column can be queried.
|
See Also:
For an example of creating an Engine preference, see "Creating an Engine Preference" in Chapter 9, "Setting Up and Managing Text". |
ConText provides the following Engine Tiles:
The ENGINE NOP Tile specifies that no engine is used for indexing. This Tile is currently not used and should not be used to create preferences for indexing.
The GENERIC ENGINE Tile specifies that the engine provided by ConText is used for indexing. ConText supplies a single engine that creates index entries for Context indexes, independent of the format, location, language, and character set of the text.
In particular, the GENERIC ENGINE Tile attributes specify the amount of memory allocated for indexing, and the tablespace(s) and creation parameters for the database tables and indexes that constitute a ConText index.
|
See Also:
For descriptions of the ConText index tables and indexes, see "Appendix C, "ConText Index Tables and Indexes". |
GENERIC ENGINE has the following attribute(s):
index_memory specifies the amount of memory, in bytes, allocated for indexing.
optimize_default specifies the type of optimization used when CTX_DDL.OPTIMIZE_INDEX is called without an optimization type. If no value is specified for optimize_default, the default is DEFRAGMENT_TO_TWO_TABLE.
i1t_tablespace, ktb_tablespace, and lst_tablespace specify the tablespaces used for the ConText index tables created during indexing.
sqr_tablespace specifies the tablespace used for the stored query expression result (SQR) table that is created, but not populated, during indexing. The SQR table for a policy stores the results of stored query expressions for the policy.
i1i_tablespace, kid_tablespace, kik_tablespace, and lix_tablespace specify the tablespaces used for the Oracle indexes generated for each ConText index table.
sri_tablespace specifies the tablespace used for the Oracle index generated for each SQR table.
xxx_storagei1t_storage, ktb_storage, and lst_storage specify the STORAGE clauses used to create the ConText index tables during ConText indexing.
sqr_storage specifies the STORAGE clause used to create the stored query expression result (SQR) table during ConText indexing.
i1i_storage, kid_storage, kik_storage, and lix_storage specify the STORAGE clauses used to create the Oracle indexes for each ConText index table.
sri_storage specifies the STORAGE clause used to create the Oracle index for each SQR table.
|
See Also:
For more information about the STORAGE clause, see the CREATE TABLE and CREATE INDEX commands in Oracle8 Server SQL Reference. |
i1t_other_parms, ktb_other_parms, and lst_other_parms specify any additional parameters used to create the ConText index tables during ConText indexing.
sqr_other_parms specifies any additional parameters used to create the stored query expression result (SQR) table during ConText indexing.
i1i_other_parms, kid_other_parms, kik_other_parms, and lix_other_parms specify any additional parameters used to create the Oracle indexes for each ConText index table.
sri_other_parms specifies any additional parameters used to create the Oracle index for each SQR table.
|
Note: In particular, the xxx_other_parms attributes are used to specify a value for the PARALLEL clause in the CREATE TABLE|INDEX command. The PARALLEL clause determines the degree of parallelism used by the Oracle parallel query option for operations such as generating Oracle indexes. For more information about the PARALLEL clause in CREATE TABLE and CREATE INDEX, as well as the other parameters that can be used to create database tables and indexes, see Oracle8 Server SQL Reference. For more information about the parallel query option in Oracle, see Oracle8 Server Tuning. |
|
See Also:
For more information about SQEs, see Oracle8 ConText Cartridge Application Developer's Guide. |
The following example creates a preference named doc_engine for the GENERIC ENGINE Tile:
begin ctx_ddl.set_attribute ('INDEX_MEMORY', 30000000 ); ctx_ddl.set_attribute ('I1T_TABLESPACE', 'DOCUMENTS' ); ctx_ddl.set_attribute ('I1T_STORAGE',' initial 10M next 2M maxextents 10'); ctx_ddl.set_attribute ('I1T_OTHER_PARMS',' pctfree 20'); ctx_ddl.set_attribute ('I1I_OTHER_PARMS',' parallel 2'); ctx_ddl.create_preference ('DOC_ENGINE', 'Test case', 'GENERIC ENGINE' ); end;
The Tiles in the Wordlist category are used to create preferences for enabling three of the ConText query expansion methods:
Stemming expands a query by deriving variations (verb conjugation, noun, pronoun, and adjective inflections) of the search token(s) in the query.
For example, a stem search on the verb buy expands to include its alternate verb forms, such as buys, buying, and bought, but not on the noun buyer. A search on the noun buyer would expand only to include its plural form buyers.
Since different languages have different stemming rules, stemming is language-dependent and uses term lists that define the relationships between the words in a given language
ConText provides a stemmer, licensed from Xerox Corporation, that utilizes Xerox Lexical Technology to support inflectional and derivational stemming in English and inflectional stemming in a number of Western European languages.
Fuzzy matching expands queries by including terms that are spelled similar to the search token in the query. This type of expansion can be useful in queries for text that contains frequent misspellings or has been scanned using OCR software.
For example, a fuzzy matching query for the term cat expands to include cats, calc, case.
The number of expansions generated by fuzzy matching depends on the tokens that ConText identified during indexing; results can vary significantly according to the tokens that were identified and indexed by ConText for the column. As such, fuzzy matching depends on how tokens are delimited in a given language.
During text indexing of a column, Soundex, if enabled, creates a list of all the words that sound alike and assigns one or more IDs to each word to identify the other words in the list that sound like the word.
The Soundex wordlist is stored in the DR_nnnnn_I1W ConText index table, where nnnnn is the identifier of the policy for the text index.
If Soundex is enabled for a text column, users can call Soundex in a query to expand the query. Soundex expands a query by searching the I1W table for terms that sound similar to the specified query term.
For example, a Soundex search on the name Smith would also find the names Smythe and Smit.
|
Note: Soundex in ConText uses the same algorithm as the SOUNDEX function in SQL. For more information about the SOUNDEX function in SQL, see Oracle8 Server SQL Reference. |
ConText provides the following Wordlist Tiles:
The GENERIC WORD LIST Tile is used to specify the advanced query options for ConText indexes. ConText provides a single Tile for handling stemming, fuzzy matching, Soundex, and named section searching.
GENERIC WORD LIST has the following attribute(s):
The stclause attribute specifies the STORAGE clause used to create the Soundex wordlist table during ConText indexing. The Soundex wordlist table is only created if Soundex is enabled through the soundex_at_index attribute.
The instclause attribute specifies the STORAGE clause used to create the Oracle index for the Soundex wordlist table.
The soundex_at_index attribute specifies whether ConText generates Soundex word mappings and stores them in the Soundex wordlist table during text indexing. If Soundex word mappings are not generated and stored in the wordlist table during indexing, queries that use Soundex are not expanded.
The stemmer attribute specifies the stemmer used for word stemming in text queries. For all the supported languages, the stemmers return standard inflected forms of a word, such as the plural form (e.g. department --> departments).
For English, an additional stemmer is provided which returns standard inflected forms and derived forms (e.g. department --> departments, departmentalize).
The default for stemmer is 1 (inflectional English)
The fuzzy_match attribute specifies which fuzzy matching routines are used for the column. Fuzzy matching is currently supported for English, Japanese, and, to a lesser extent, the Western European languages.
The default for fuzzy_match is 1.
section_groupThe section_group attribute specifies the name of the section group to assign to a text column. The following rules apply to section_group:
|
See Also:
For more information about section groups, see "Document Sections" in Chapter 6, "Text Concepts". |
The following example creates a preference named soundex_yes for the GENERIC WORD LIST Tile:
begin ctx_ddl.set_attribute('SOUNDEX_AT_INDEX', '1'); ctx_ddl.create_preference('SOUNDEX_YES', 'Will build the soundex mapping during indexing', 'GENERIC WORDLIST'); end;
The Stoplist Tiles are used to create Stoplist preferences. A stoplist is a list of common terms that ConText does not include in the text index for a text column.
Each stoplist can contain a maximum of 4095 words.
|
See Also:
For an example of creating a Stoplist preference, see "Creating a Stoplist Preference" in Chapter 9, "Setting Up and Managing Text". |
The Stoplist category contains the following Tiles:
| Tile | Attributes | Attribute Values |
|---|---|---|
|
GENERIC STOP LIST |
stop_word |
word (string), sequence (number) |
The GENERIC STOP LIST Tile specifies the terms that should not be included in the text index.
GENERIC STOP LIST has the following attribute(s):
The stop_word attribute has two values that must be specified:
sequence is a value from 1 to 4095 and is used in a text index to record the stop words that proceed and follow an indexed term. ConText records up to eight preceding stop words and eight following stop words for each indexed term. This enables text queries for phrases which contain stop words.
For example, consider the sentence "he is at the top of the class" where at, the, top, and of are stop words. The sequences for each of the stop words are recorded as part of the text index entry for the term class, which allows users to include stop words in a query (e.g. 'top of the class').
The following example creates a preference named mini_stoplist for the GENERIC STOP LIST Tile:
begin ctx_ddl.set_attribute ('STOP_WORD', 'a', 1); ctx_ddl.set_attribute ('STOP_WORD', 'A', 2); ctx_ddl.set_attribute ('STOP_WORD', 'the', 3); ctx_ddl.set_attribute ('STOP_WORD', 'The', 4); ctx_ddl.set_attribute ('STOP_WORD', 'and', 5); ctx_ddl.set_attribute ('STOP_WORD', 'And', 6); ctx_ddl.create_preference ('MINI_STOPLIST', 'minilist', 'GENERIC STOP LIST' ); end;
