<commit_before>---
layout: post
title: Graph Theory Basics, and Speech Recognition with Neural Nets
---

Here are the highlights from this week's ["Talking Machines"](http://www.thetalkingmachines.com/) on twitter at (). Thank you [thunder](https://github.com/ThunderShiviah/) for turning me on to this awesome podcast! This is a work in progress, so bear with me. The guests talked fast and used a lot of acronyms.

Graphs
------

<NAME> and <NAME> talked about graph theory and approaches to selecting a model for a given Data Science problem. Kyle recommends trying off-the-shelf implementations of simple algorithms like Logistic Regression, Support Vector Machines, or Random Forests. Graph Limits are an interesting area of research, for dealing with large networks.

You can then branch your decision tree as you discover these two main classes of "difficult problems"

1. Logistic Regrssion will fail on big nonlinearities.
1. a. just add nonlinearities (kernels or transformations) to your LR, SVM
1. b. use neural nets or other nonlinear techniques
2. High dimensionality, has two solutions
2. a. Image processing doesn't reduce the dimensions, just uses convolutional neural nets to segment the dimensions
2. b. Genetics -- shrink most weights down to zero to find strongest influencers 

Probabilistic matrix factorization and SVD are both used for dimension reduction and latent feature identification.

Neural Nets for Speech
----------------------

In the second half they interviewed <NAME> who shook up the world of speech recognition by employing deeper neural nets in place of Gaussian Mixture Models to solve the Merck problem.


[Kaldi](http://kaldi.sourceforge.net/about.html) has recently incorporated neural nets. So we have an open source project to use for state-of-the art LVCSR research.

[<NAME>](http://www.cs.toronto.edu/~asamir/) and [<NAME>](http://www.cs.toronto.edu/~gdahl/) at Torronto worked on phoenem recognition on [TIMIT](https://en.wikipedia.org/wiki/TIMIT)
<NAME> at Microsoft understood TIMIT and hired Abdur and George to try their technique on large vocabularies (Bing searches)

"The biggest problems with the current approach are the GMM, HMM, N-gram language model" (everything except the beam search decoder part of the pipeline which is pretty good).

Bioinformatics ignored neural nets until <NAME> and Ruslan (R<NAME> won the Merck challenge on Kaggle.
<NAME>ena 15% improvement on baseline by Merck
Bayesean Neural Nets useful on large datasets as opposed to conventional multilayer perceptron Neural Nets.

Alphabet Soup
-------------
LR: Logistic Regression
SVM: Support Vector Machines
FMLLR: ? ? Linear Logistic Regression
VMMI: ?
GMM: Gaussian Mixture Model, first step in speech recognition, creates training set
HMM: Hidden Markhov Model, N-gram language model, 
MCMC: Markov Chain Monte Carlo
MCRBM: Monte Carlo Restricted Boltzman Machine
[TIMIT](https://en.wikipedia.org/wiki/TIMIT): Texas Instruments and Mass. Inst. Tech, speech recognition data set and problem
LVCSR: Large Vocabulary Continuous Speech Recognition
NNLM: Neural Net Language Models
RNN: Recurrent Neural Net
QSAR: Quantitative Structural Activity Relation prediction (the Merck kaggle challenge)
GBRBM: Gaussian Bernouli Restricted Boltzman machine

People
------
Dong Yu fixed hard-coded 1250 limit on 4-bit symbols in the encoder.
<NAME> -- network theory
<NAME> -- went to University of Toronto  
Geoffrey (Geoff) Hinton -- researcher at University of Toronto http://www.cs.toronto.edu/~hinton
<NAME> -- using RNN for for a language model
<NAME> -- colaborated with Geof and <NAME> on the Merck challenge
[<NAME>](http://www.cs.toronto.edu/~asamir/) -- collaborator with George Dahl on neural nets in speech recognition

<commit_msg>capitalize Thunder<commit_after>---
layout: post
title: Graph Theory Basics, and Speech Recognition with Neural Nets
---

Here are the highlights from this week's ["Talking Machines"](http://www.thetalkingmachines.com/) on twitter at (). Thank you [Thunder](https://github.com/ThunderShiviah/) for turning me on to this awesome podcast! 

The guests talked fast and used a lot of acronyms so bear with me.

Graphs
------

<NAME> and <NAME> talked about graph theory and approaches to selecting a model for a given Data Science problem. Kyle recommends trying off-the-shelf implementations of simple algorithms like Logistic Regression, Support Vector Machines, or Random Forests. Graph Limits are an interesting area of research, for dealing with large networks.

You can then branch your decision tree as you discover these two main classes of "difficult problems"

1. Logistic Regrssion will fail on big nonlinearities.
1. a. just add nonlinearities (kernels or transformations) to your LR, SVM
1. b. use neural nets or other nonlinear techniques
2. High dimensionality, has two solutions
2. a. Image processing doesn't reduce the dimensions, just uses convolutional neural nets to segment the dimensions
2. b. Genetics -- shrink most weights down to zero to find strongest influencers 

Probabilistic matrix factorization and SVD are both used for dimension reduction and latent feature identification.

Neural Nets for Speech
----------------------

In the second half they interviewed George Dahl who shook up the world of speech recognition by employing deeper neural nets in place of Gaussian Mixture Models to solve the Merck problem.


[Kaldi](http://kaldi.sourceforge.net/about.html) has recently incorporated neural nets. So we have an open source project to use for state-of-the art LVCSR research.

[<NAME>](http://www.cs.toronto.edu/~asamir/) and [<NAME>](http://www.cs.toronto.edu/~gdahl/) at Torronto worked on phoenem recognition on [TIMIT](https://en.wikipedia.org/wiki/TIMIT)
Lee Dang at Microsoft understood TIMIT and hired Abdur and George to try their technique on large vocabularies (Bing searches)

"The biggest problems with the current approach are the GMM, HMM, N-gram language model" (everything except the beam search decoder part of the pipeline which is pretty good).

Bioinformatics ignored neural nets until <NAME> and Ruslan (<NAME> won the Merck challenge on Kaggle.
<NAME> 15% improvement on baseline by Merck
Bayesean Neural Nets useful on large datasets as opposed to conventional multilayer perceptron Neural Nets.

Alphabet Soup
-------------

LR: Logistic Regression
SVM: Support Vector Machines
FMLLR: ? ? Linear Logistic Regression
VMMI: ?
GMM: Gaussian Mixture Model, first step in speech recognition, creates training set
HMM: Hidden Markhov Model, N-gram language model, 
MCMC: Markov Chain Monte Carlo
MCRBM: Monte Carlo Restricted Boltzman Machine
[TIMIT](https://en.wikipedia.org/wiki/TIMIT): Texas Instruments and Mass. Inst. Tech, speech recognition data set and problem
LVCSR: Large Vocabulary Continuous Speech Recognition
NNLM: Neural Net Language Models
RNN: Recurrent Neural Net
QSAR: Quantitative Structural Activity Relation prediction (the Merck kaggle challenge)
GBRBM: Gaussian Bernouli Restricted Boltzman machine

People
------

<NAME> fixed hard-coded 1250 limit on 4-bit symbols in the encoder.
<NAME> -- network theory
<NAME> -- went to University of Toronto  
Geoffrey (<NAME> -- researcher at University of Toronto http://www.cs.toronto.edu/~hinton
<NAME> -- using RNN for for a language model
<NAME> -- colaborated with Geof and <NAME> on the Merck challenge
[<NAME>](http://www.cs.toronto.edu/~asamir/) -- collaborator with <NAME> on neural nets in speech recognition


{'fraction_non_alphanumeric': 0.07765050612679808, 'fraction_numerical': 0.0034629728289824187, 'mean_word_length': 4.417748917748917, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 6, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5048172', 'n_tokens_mistral': 2107, 'n_tokens_neox': 1929, 'n_words': 996}