Tuesday, November 1, 2011

IR Assignment 2.

Hi all.

Once more, something completly abstract and incoherent... This is way too thoretical, for even the practical part to make any sence.
So, in a few words to wrap-up:

"What the hell are we supposed to do?"

Sentences like "It is up to you to come up with a way of doing this" or "the data gives you a lot of freedom as to what you want to look at in this assignment" or "you can try very different experiments to get there" or "and much, much more".

You know, I am used to having to do something under strict specifications and explicit requirements.
e.g. Use this. Do that. Find 1, 2, 3. Build 4, 5, 6.

The endless freedom of specifications and the randomly seeming resources given for this assignment, really frustrate me... I need to work in something solid!

7 comments:

  1. Get used to it. In real life things don't come with clear specifications and explicit requirements.

    There are many approaches you could take for this exercise. The most straightforward would be to just retrieve a set of documents for your query, then for each expert, look at the ranks of the documents associated with that expert and calculate a score from that list of ranks. You can then rank the experts by their score.

    ReplyDelete
  2. As Taco said, the world of academia doesn't really do strict specifications. This is science, not business.

    The point of the assignment is:

    1. To build a system that does expert retrieval (ie given a query find which experts are relevant to the query).

    2. To come up with a hypothesis (relating to the system above) that you can then test (by doing experiments) to see whether it is true or false. The A, B, C, D in the assignment explanation are just examples of what kind of hypothesis you might use.

    ReplyDelete
  3. I figured out a basic "skeleton" to solve this assignment:

    0. Define what you want to investigate (eg. "I will combine the results from the three different collections provided, looking for their optimal wheighting.")

    1. Index the documents provided. I made 3 indices, in 3 different folders (index.body/, index.title/, index.metadata/), in order, later, to retrieve and integrate 3 different result lists.

    2. Query your index/indices, retriving the results for the topics provided by the teacher. At this point you can evaluate how good your index is, comparing the results with the ground truth (document-qrels.txt) using, for instance, trec_eval.

    3. Perform result integration. If you have different indices, and thus more than one result list, you have to produce an unique list. Here you have to solve two tasks: 1) Define the criteria you'll use to perform the integration (how you define the final score? What happens if a document in list A is not present in list B? How much wheight you'll give to each result list) 2) Use a programming language to perform the integration. Personally I'll use Python, because it is fairly simple and has a good support for regular expressions, which can help you a lot to parse the result files (check this out: http://rubular.com/r/3mNrg5eqph).

    4. Associate the experts to the documents. Once you have your final document list, you'll have to associate the document IDs with the relative email addresses (associations.txt).

    5. Produce a trec-formatted expert list.

    6. Evaluate the list.

    7. Write the report :-)


    Hoping not to have written idiocies :-)

    ReplyDelete
  4. Dario, it's just one collection. The three files represent different fields in the collection. If I understand you right, mixing up results from body and title should be fine, but I think the metadata is just empty (at least as far as I saw).

    ReplyDelete
  5. Yeah you're right: I used the term "collection" improperly. Still, it make sense to produce one index per field right?

    The metatata are not empty, check for instance "corpus.metadata/CISRO229". The problem is that both the three fields, and especially "metadata" and "title" contain a big number of empty documents, so we have to find a way to integrate the 3 sources without penalizing the absence of a document one to another list..

    ReplyDelete
  6. Not entirely sure about this, but I think INDRI supports fields in the index. Take a look at the IndriBuildIndex docs. Even if its possible, itmight still be desirable or interesting to try to do it with seperate indices.

    ReplyDelete
  7. Thanks for all your answers. I think I just don't get some parts though. Anyway. I'll try to "suspect" (with, oh yes, full sarcasm) what the first decision is all about. (And that's because there are plenty (many many ways?!) to do this!)

    And what about "headers"? Nobody mentioned headers. I think I'll treat 'em all together.
    One big collection. Boom, body+title+metadata+header. As if I know what each shit is and as the explanations they give mean something. Mercy.

    Above all, we also have those texts with this damned linux style EOL thing. As a DOS guy I just WONDER why they couldn't just add both EOL chars. Was that TOO difficult for them? Easy, CR+LF. Now, I have to work-around string processing...

    ReplyDelete