py4tfidf
Term Frequency – Inverse Document Frequency (TF-IDF) Python Library
Getting Started
This project is simply an implementation of TF-IDF algorithm in python programming language.
Prerequisites
Numpy
Installing
The easiest way to install py4tfidf is by using pip
pip install py4tfidf
Usage
There are 2 public methods of Tfidf
class. It is vectorize_train
and vectorize_test
. vectorize_train
used to build the corpus, calculate idf based on training text, and transform it into a usable vector by multiplying its tf and its idf, while vectorize_test
is just simply transforming the test text into a usable vector by multiplying its tf with previously obtained if. Both vectorize_train
and vectorize_test
take 1 argument namely x_train and x_text respectively. Because tokenizing is usually done in the text preprocessing phase, we assume you tokenize your text on your own, so the argument for vectorize_train
and vectorize_test
should be a list of tokenized text.
from py4tfidf.vectorizer import Tfidf
vec = Tfidf()
x_train = [['i','love', 'python'],['natrual','language','processing','is','fun'],['python','is','fun']]
x_test = [['python','language','is','fun'],['im','learning','natrual','language','processing']]
x_train = vec.vectorize_train(x_train)
x_test = vec.vectorize_test(x_test)