I wanted to started experimenting with neural network and as a toy problem I wished to train one to chat, i.e. implement a chatting bot like cleverbot. Not that clever anyway.I looked around for some documentation and I found many tutorial on general tasks, but few on this specific topic. The one I found just exposed the results without giving insights on the implementation. The ones that did, did it pretty shallowy (the tensorflow documentation page on seq2seq is lacking imho).Now, I feel I may have understood the principle more or less but I'm not sure and I am not even sure how to start. Thus I will explain how I would tackle the problem and I'd like a feedback on this solution, telling me where I'm mistaken and possibly have any link to detailed explainations and practical knowledge on the process.The dataset I will use for the task is the dump of all my facebook and whatsapp chat history. I don't know how big it will be but possibly still not large enough. The target language is not english, therefore I don't know where to quickly gather meaningful conversation samples.I am going to generate a thought vector out of each sentence. Still don't know how actually; I found a nice example for word2vec on deeplearning4j website, but none for sentences. I understood how word vectors are built and why, but I could not find an exhaustive explaination for sentence vectors.Using thought vectors as input and output I am going to train the neural network. I don't know how many layers it should have, and which ones have to be lstm layers.Then there should be another neural network that is able to transform a thought vector into a sequence of character composing a sentence. I read that I should use padding to make up for different sentence lengths, but I miss how to encode characters (are codepoints enough?).
How to train a chatbot
neural networks;chat bots;recurrent neural networks;lstm
I would recommend to start by reading this blogpost.You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement.That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to concern you.The next more complex step would be to input word vectors instead of characters. That would allow you to generalise to words that aren't part of your training data. And it is probably still just a minor modification of the code. If you insist on using thought vectors, you should start reading up on NN translation. And probably try to get a pre-trained encoder network. Or pre-train it yourself on a large translation corpus for your language. With your small training set the best you can do is probably massively overfit until your system recreates your training data verbatim. Using word vectors will allow your system to give the same answer to I beat the cat today. as you gave in the training data to I kicked the dog yesterday. I'm not sure thought vectors will make a big difference. If you get the decoder to learn at all. 