Add get position list function #223
Conversation
| @@ -72,6 +72,15 @@ print(doc_vector) | |||
| ``` | |||
|
|
|||
| The result is a dictionary where the keys are the analyzed terms and the values are the term frequencies. | |||
|
|
|||
| If you want to know the positions of each term in the document, you can use `get_term_positions`: | |||
There was a problem hiding this comment.
"of each term" -> "of every term"?
docs/usage-indexreader.md
Outdated
| print(term_positions) | ||
| print(indexed_doc) | ||
| ``` | ||
| The result is a tuple. The first member is a dictionary where the keys are the analyzed terms and the values are the positions each term occur in the document. The second member is a string containing the recovered document content using the position information. |
There was a problem hiding this comment.
Here I think you can just write Tuple[Dict[str, int], str] - a Python programmer should be able to interpret type signatures, and it's more concise.
pyserini/index/_base.py
Outdated
| ------- | ||
| Optional[Tuple[Dict[str, int], str]] | ||
| A tuple contains a dictionary with analyzed terms as keys and corresponding posting list as values, and a | ||
| string representing the recovered document |
There was a problem hiding this comment.
Why do you want to return "string representing the recovered document"? What's the use case? Users can easily do this also if they want?
There was a problem hiding this comment.
Currently, the document returned by doc.contents() is the document before it is processed by Lucene. The reconstructed document will provide a view of the document after Lucene's processing.
Yes. Users can easily do this if they want, and not every user needs this. I think we can move this piece of code to usage-indexreader.md as an example of how to use this function.
expose getTermPositions in the IndexReader class. Aside from the term positions mapping, this function also returns a reconstructed document using the mapping. This should help users to the effect of stop words removal and stemming. Currently, the document returned by doc.contents() is the document before it is processed by Lucene. The reconstructed document will provide a view of the document after Lucene's processing.