Jai Ho

Assignment Inverted indicant October 19, 2012 1 Introduction Today, top look to engines the like Google and Yahoo part a data structure called Inverted index finger for their matching of queries to the muniments and work users the relevant memorials jibe to their rank. Inverted Index is basically a map from a volume to its position of occurence in the document. Since a word may appear more than once in the document, storing all the positions and the frequency of a word in the document gives an idea of relevancy of this document for a particular word.If such an anatropous index is build up for individually document in the collection, whence when a query is ? red, a depend can be done for the query in these indexes and ranking is obtained according to the frequency. Mathematically, an upside-down index for a document D and thread s1 , s2 , , sn is of the found s1 ? a1 , a1 , 1 2 s2 ? a2 , a2 , 1 2 . . . sn ? an , an , 2 1 where ak denotes the lth position of k th word in the document D. l To build up this kind of data structure e? iently, Tries are utilise. Tries are a good data structure for strings as searching becomes very simple here with either ruffle node describing one word. To build up an change index accustomed a set of documents using trie, following go are followed breed one document and insert words into a trie. As a leaf node is reached, assign it a number (in increasing order) representing its muddle in the index (staring from 0). Add the position of this word into the index. Now for a word which occur more than once in the document, when attempt for heartbeat insertion into the trie is made, a leaf node already containing that word would be found and its value would tell the location in the index. So entirely go to this index and add another position for this word. Do this cashbox end of document is reached. Now, you sop up a trie and an inverted index for the ? rst document. bear this procedure for the rest of the documents. 1 Now follow the below steps to search for a word from the inverted indexes and tries of all the documents For every document, ? st search for the word in the match trie and get its location in the inverted index of that document. Then traverse through all the positions and see which document has most frequency and arrange the documents accordingly (in fall order). Also, in every document there are special words called anchor texts which have more importance than a frequent text word. For example a download link. So for the same word, its occurence as an anchor text increases the relevance of that document over its normal occurence. 2 Problem StatementFor this assignment, you need to realize an inverted index for a collection C of documents from 1 to n. all document willing be a plain text ? le with ? rst line storing its id from 1 to n and next few lines containing space or new line separated words. The index should be an array of lists with surface of array equal to total number of distinct words in the array and the list for each word contains the locations of the word in the document. The trie used for this construction can be represented in any form (array/linked list/trees etc. ).So you would have n such tries and inverted indexes. Then you should ask user for the queries (single-word) and give the order of documents in decreasing order of relevance. For our case, the anchor texts are represented by following the word with a ?. So if you have something like Rats fear cats and cats* fear dogs. then here 1st cat is a normal word whereas second cat is an anchor text. So now your array size will be 2 ? totalnumberof distinctwords in the document as you would store positions of normal text and anchor text separately for a given word.And now relevance should ? rst be decided by the frequency of anchor texts and in spite of appearance them collision should be resolved by frequency of normal text. D1 D2 D3 1 it is what it is 2 wh at is it 3 it is a banana Below are the tally tries and inverted indexes for the 3 documents (? gure 1). 2 Figure 1 Trie and Inverted Index for Documents 1, 2 and 3 Now if query is it then search in 1st index gives 0, 3(f req = 2), 2nd index gives 2(f req = 1) and 3rd one gives 0(f req = 1).So, our widening is 1, 2, 3or1, 3, 2 (as document 2 and 3 have equal relevance). line of credit The names of the data ? les should be taken from command line. After 3 building the inverted index, you should ask for query again from command pep up and also give an option of quitting any time the user want. The inverted indexes should be written to ? les named as 1 n. txt with each line corresponding to one word in the document. You can ignore case-sensitive words i. e. , draw and cat are same. Also ignore symbols in the text (if any) like . ,-? 4