While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation? 
What is the difference between word-based and char-based text generation RNNs?
machine learning;neural network;nlp;rnn
Here is what I learnt recently.Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).Word-based LMs display higher accuracy and lower computational cost than char-based LMs. This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.Therefore, we can say that one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:The surprised in investors werent going to raise money. Im not the company with the time there are all interesting quickly, dont have to get off the same programmers.While simple char-based Maximum Likelihood LM with a 13-character window delivers this:And when she made many solid bricks. He stacked them in piles and stomped her  feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked  her out.Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.