An Information-Theoretic Approach to
Text Searching in Direct Access Systems

Using direct access computer files of bibliographic
information, an attempt is made to overcome 
one of the problems often associated with information
retrieval, namely, the maintenance and use of large 
dictionaries, the greater part of which is used only
infrequently.  A novel method is presented, which 
maps the hyperbolic frequency distribution.  This is
more suited to implementation on storage devices. 
 This method treats text as a string of characters rather
than words bounded by spaces, and chooses subsets 
of strings such that their frequencies of occurrence are
more even than those of word types.  The members 
of this subset are then used as index keys for retrieval.
The rectangular distribution of key frequencies 
results in a much simplified file organization
and promises considerable cost advantages.

CACM June, 1974

Barton, I. J.
Creasey, S. E.
Lynch, M. F.
Snell, M. J.

text searching, information theory, file organization,
direct access, information retrieval, character 
string, bit vector

3.42 3.70 3.73 3.74 5.6

CA740615 JB January 17, 1978  2:07 PM

2532	4	2631
2631	4	2631
2631	4	2631
2746	4	2631
3001	4	2631
2139	5	2631
2532	5	2631
2631	5	2631
2631	5	2631
2631	5	2631