Comments on: A quick foray into linear algebra and Python: tf-idf http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/ A tagline? What am I…a super hero? Fri, 15 May 2009 23:33:19 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-384 Tim Trueman Wed, 20 Aug 2008 01:21:07 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-384 @Thomas, glad you found it useful! @Thomas, glad you found it useful!

]]> By: Thomas http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-379 Thomas Tue, 19 Aug 2008 09:58:47 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-379 Wonderful post, it makes a technical concept from Natural Language Processing easily understandable. Hurrah for pseudocode, here's to Python. Wonderful post, it makes a technical concept from Natural Language Processing easily understandable. Hurrah for pseudocode, here’s to Python.

]]> By: Fitzgerald Steele http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-338 Fitzgerald Steele Thu, 14 Aug 2008 18:45:22 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-338 Thanks for the simple explaination of TF-IDF. Joséphine, and others...reading from a file in python is pretty straightforward. Let's assume we're using Python 2.5+ <code> f = open('file1.txt') txt = f.read() f.close() documentList.append(txt) Of course, we should put some error handling around it using a try/except, or the with statement if you're using Python 2.5 or better. Thanks for the simple explaination of TF-IDF.

Joséphine, and others…reading from a file in python is pretty straightforward. Let’s assume we’re using Python 2.5+


f = open('file1.txt')
txt = f.read()
f.close()
documentList.append(txt)

Of course, we should put some error handling around it using a try/except, or the with statement if you're using Python 2.5 or better.

]]> By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-336 Tim Trueman Wed, 13 Aug 2008 16:52:47 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-336 Joséphine, What you are trying to do is beyond the scope of this tutorial. However, you can find out how to read from a file with a simple search query. Joséphine,

What you are trying to do is beyond the scope of this tutorial. However, you can find out how to read from a file with a simple search query.

]]> By: Joséphine http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-335 Joséphine Wed, 13 Aug 2008 12:11:00 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-335 Hi Tim, my purpose ist to compare wikipedia articles with wordnet words in order to find wich wikipedia article the right one to the wordnet word. Thanks in advance for your answer. Hi Tim,
my purpose ist to compare wikipedia articles with wordnet words in order to find wich wikipedia article the right one to the wordnet word.
Thanks in advance for your answer.

]]> By: Joséphine http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-334 Joséphine Wed, 13 Aug 2008 12:08:23 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-334 HI Tim, You code ist very great! But I have a question, how can I use it by reading from file? thanks HI Tim,

You code ist very great!
But I have a question, how can I use it by reading from file?
thanks

]]> By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-110 Tim Trueman Mon, 30 Jun 2008 20:19:00 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-110 KARL, This code doesn't read from a file at all, in order to keep it simple and avoid unnecessary code I just made it use strings as if they were separate documents. The way to use this code is to replace <code>”"”DOCUMENT #1 TEXT”"”</code> with your space-delimited document inside the three quotes (e.g. <code>"""The quick brown fox jumped over the lazy dog"""</code>). Do this for each document you are using. KARL,

This code doesn’t read from a file at all, in order to keep it simple and avoid unnecessary code I just made it use strings as if they were separate documents.

The way to use this code is to replace ”"”DOCUMENT #1 TEXT”"” with your space-delimited document inside the three quotes (e.g. """The quick brown fox jumped over the lazy dog"""). Do this for each document you are using.

]]> By: KARL http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-109 KARL Sun, 29 Jun 2008 10:31:32 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-109 Hi Tim thanks for this code - need to do term frequency calc for entries in a database. not really that familiar with python. If i have a space separated file of words - where in the code above do i put the file name? thanks Hi Tim

thanks for this code – need to do term frequency calc for entries in a database.
not really that familiar with python. If i have a space separated file of words – where in the code above do i put the file name?
thanks

]]> By: m13 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-106 m13 Fri, 20 Jun 2008 10:53:26 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-106 yes, currently I am writing my own tokenizer that will be applied before feature selection/weighting, that's why I needed to understand whether this code includes (simple) tokenization or not (as I already mentioned, I dont know python that well) ;) thank you so much for replying in such a short time! yes, currently I am writing my own tokenizer that will be applied before feature selection/weighting, that’s why I needed to understand whether this code includes (simple) tokenization or not (as I already mentioned, I dont know python that well) ;)

thank you so much for replying in such a short time!

]]> By: Tim Trueman http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/comment-page-1/#comment-105 Tim Trueman Thu, 19 Jun 2008 21:43:56 +0000 http://timtrueman.com/2008/02/10/a-quick-foray-into-linear-algebra-and-python-tf-idf/#comment-105 Glad you've found it somewhat useful! <code>document.split(None)</code> takes the document and splits it into an array containing each word in the document (None means use spaces as the separator). This is effectively a simple tokenizer. A more sophisticated tokenizer would provide better results perhaps but for the example I decided it was good enough. Does that answer your question(s)? Glad you’ve found it somewhat useful! document.split(None) takes the document and splits it into an array containing each word in the document (None means use spaces as the separator). This is effectively a simple tokenizer. A more sophisticated tokenizer would provide better results perhaps but for the example I decided it was good enough.

Does that answer your question(s)?

]]>