This note is to log the development process of the overall system. So let's begin
The idea is we are going to build a system which can read any pdf and build a profile using vectorizer like count vectorizer this will make our system capable of understanding what kind of words are there in such document type and hence will be able to predict any new document as one of the previous datatypes
Initially we train our model to understand datas from past which are labeled as particular topic realted or not. Then we are going to use various unknown data for inference testing and see how well it performs.
Our system as of now works only with pdf documents but we can make it adaptable to work with any datatype provided we can read the text from that particular file.
We are going to take a bunch of words convert it into dataset having words and title such that the first column of dataste consists all the words required for the particular task of classification and title consists all the label assigned to a particular class.
- Libraries used
First we are going to use Preprocess to create folders where all files are stored then we are going to use Dataset Generator to generate an excell file with all the details that we are going to use for training purpose. Finally we are going to use classifier.py for classification purpose of the files
This bag of words approach can be swtiched with tfidf approach to test for new possible words and working on how it can be used to make a better classifcation model for special terms and words
This project can be scaled up to include deep learning approach to have feature extraction in and of itself.
Documentation on Progress !!