Comments on: A quick foray into linear algebra and Python: tf-idf http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/ Everything tech... Mon, 07 Jul 2008 10:04:45 +0000 http://wordpress.org/?v=2.5.1 By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-110 Tim Trueman Mon, 30 Jun 2008 20:19:00 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-110 KARL, This code doesn't read from a file at all, in order to keep it simple and avoid unnecessary code I just made it use strings as if they were separate documents. The way to use this code is to replace <code>”"”DOCUMENT #1 TEXT”"”</code> with your space-delimited document inside the three quotes (e.g. <code>"""The quick brown fox jumped over the lazy dog"""</code>). Do this for each document you are using. KARL,

This code doesn’t read from a file at all, in order to keep it simple and avoid unnecessary code I just made it use strings as if they were separate documents.

The way to use this code is to replace ”"”DOCUMENT #1 TEXT”"” with your space-delimited document inside the three quotes (e.g. """The quick brown fox jumped over the lazy dog"""). Do this for each document you are using.

]]> By: KARL http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-109 KARL Sun, 29 Jun 2008 10:31:32 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-109 Hi Tim thanks for this code - need to do term frequency calc for entries in a database. not really that familiar with python. If i have a space separated file of words - where in the code above do i put the file name? thanks Hi Tim

thanks for this code - need to do term frequency calc for entries in a database.
not really that familiar with python. If i have a space separated file of words - where in the code above do i put the file name?
thanks

]]> By: m13 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-106 m13 Fri, 20 Jun 2008 10:53:26 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-106 yes, currently I am writing my own tokenizer that will be applied before feature selection/weighting, that's why I needed to understand whether this code includes (simple) tokenization or not (as I already mentioned, I dont know python that well) ;) thank you so much for replying in such a short time! yes, currently I am writing my own tokenizer that will be applied before feature selection/weighting, that’s why I needed to understand whether this code includes (simple) tokenization or not (as I already mentioned, I dont know python that well) ;)

thank you so much for replying in such a short time!

]]> By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-105 Tim Trueman Thu, 19 Jun 2008 21:43:56 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-105 Glad you've found it somewhat useful! <code>document.split(None)</code> takes the document and splits it into an array containing each word in the document (None means use spaces as the separator). This is effectively a simple tokenizer. A more sophisticated tokenizer would provide better results perhaps but for the example I decided it was good enough. Does that answer your question(s)? Glad you’ve found it somewhat useful! document.split(None) takes the document and splits it into an array containing each word in the document (None means use spaces as the separator). This is effectively a simple tokenizer. A more sophisticated tokenizer would provide better results perhaps but for the example I decided it was good enough.

Does that answer your question(s)?

]]> By: m13 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-104 m13 Thu, 19 Jun 2008 11:36:28 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-104 hey Tim, I am not planning to use your code (besides I dont know python), I am just trying to understand it. I am doing a project reg TC and I will be using tf-idf for feature selection/weighting. From what I understand, the above code is not for tokenized text right? No matter the case, I've been searching google for 2 days now, and your post is the most helpful thing I found ;) many thanks for explaining tf-idf in terms of (pseudo)code hey Tim, I am not planning to use your code (besides I dont know python), I am just trying to understand it. I am doing a project reg TC and I will be using tf-idf for feature selection/weighting.

From what I understand, the above code is not for tokenized text right?

No matter the case, I’ve been searching google for 2 days now, and your post is the most helpful thing I found ;)

many thanks for explaining tf-idf in terms of (pseudo)code

]]>