<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "meta"
    ]
   },
   "source": [
    "# Machine learning data sets, benchmarks and competitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most famous datasets:\n",
    "* Iris data set: https://en.wikipedia.org/wiki/Iris_flower_data_set\n",
    "* MNIST database: https://en.wikipedia.org/wiki/MNIST_database\n",
    "\n",
    "List of datasets:\n",
    "* [A large list of datasets from wikipedia](https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research).\n",
    "* [UCI machine learning repository](https://archive.ics.uci.edu/ml/datasets.html): a large collection of datasets for machine learning research.\n",
    "* [Amazon datasets](https://aws.amazon.com/fr/datasets/): Public datasets hosted by Amazon.\n",
    "* [Computer vision datasets](http://www.cvpapers.com/datasets.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "Miscellaneous data:\n",
    "* https://voice.mozilla.org/ : an open source voice data set (see also http://linuxfr.org/news/mozilla-common-voice-vers-une-reconnaissance-vocale-libre)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks and competitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most famous competitions:\n",
    "\n",
    "* [Kaggle](https://www.kaggle.com/) : the most popular competition platform. Contains a large list of [data sets](https://www.kaggle.com/datasets) too. More information on [Wikipedia](https://en.wikipedia.org/wiki/Kaggle).\n",
    "\n",
    "List of competition:\n",
    "\n",
    "* http://www.chalearn.org/challenges.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "**TODO** (from http://www.chalearn.org/challenges.html):\n",
    "* Tunedit: Similar platform more academically oriented (phased out?).\n",
    "* DrivenData: For non-profit challenges.\n",
    "* Codalab: For academic challenges of greater complexity.\n",
    "* Beat: A EU sponsored platform.\n",
    "* Epidemium: challenges in epidemiology.\n",
    "* Pascal challenges: The Pascal network is sponsoring several challenges in Machine learning.\n",
    "* Challenges.gov: Challenges sponsored by the US Government.\n",
    "* Ecole Normale Superieure: Datasets and challenges.\n",
    "* Beaker notebook: Convert back and forth from R/Python/Javascript\n",
    "* Cortana Intelligence: Azure ML platform.\n",
    "* RAMP studio: The Paris-Saclay CDS Rapid Analytics Model Prototyping platform.\n",
    "* Synapse: The platform on which DREAM challenges are organized.\n",
    "\n",
    "Collaborative platforms:\n",
    "* OpenML: share ML reusable frameworks.\n",
    "* MLcomp: compare machine learning programs.\n",
    "* E-lico: data mining portal.\n",
    "* H20: open source predictive analytics platform.\n",
    "* KNIME: Data mining platform.\n",
    "* Quantopian: Financial data simulator + ML tutorials.\n",
    "\n",
    "Crowdsourcing:\n",
    "* Amazon Mechanical Turk: Gets you hire people from all around the world to solve your tasks. Used to label computer vision data.\n",
    "* Crowdflower: Hire people to collect, filter and enhance data.\n",
    "\n",
    "International conferences hosting challenges:\n",
    "* WCCI: World congress on computational intelligence.\n",
    "* ICDAR: International Conference on Document Analysis and Recognition, a bi-annual conference proposing a contest in printed text recognition. Feature extraction/selection is a key component to win such a contest.\n",
    "* ICPR: In conjunction with the International Conference on Pattern Recognition, ICPR 2004, a face recognition contest is being organized.\n",
    "* ICMI: Competitions on multimodal interaction\n",
    "\n",
    "Popular challenges:\n",
    "* NNGC: Neural Network Grand Challenge in time series forecasting.\n",
    "* Netflix: The 1 million dollar Netflix prize, which attracted a lot of attention and broke new grounds for recommender systems.\n",
    "* Robocup: Robots who play soccer, a yearly held contest.\n",
    "* DELVE: A platform developed at University of Torontoto benchmark machine learning algorithms.\n",
    "* CAMDA: Critical Assessment of Microarray Data Analysis, an annual conference on gene expression microarray data analysis. This conference includes a context with emphasis on gene selection, a special case of feature selection.\n",
    "* TREC: Text Retrieval conference, organized every year by NIST. The conference is organized around the result of a competition. Past winners have had to address feature extraction/selection effectively.\n",
    "* CASP: An important competition in protein structure prediction called Critical Assessment of Techniques for Protein Structure Prediction.\n",
    "* ICAPS competitions: Competitions in planning and knowledge engineering\n",
    "* MEDIAEVAL benchmarks: Benchmarking Initiative for Multimedia Evaluation. Data sharing in multimediacommons (with incremental annotations). Uses Amazon web services to allow experimentation in the cloud.\n",
    "* DREAM: Dialogue for Reverse Engineering Assessments and Methods. Challenges in gene network reconstruction.\n",
    "* AVEC: Audio visual Emotion Recognition Challenge and Workshop.\n",
    "* CAFA: Predicting function of biological macromolecules (as well as gene-disease associations).\n",
    "\n",
    "Data resources:\n",
    "* KEEL: Knowledge Extraction based on Evolutionary Learning.\n",
    "* IO Data Science: Datasets of Paris-Saclay University."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "**TODO**: filter the following list...\n",
    "* https://en.wikipedia.org/wiki/Competitive_programming\n",
    "* https://en.wikipedia.org/wiki/American_Computer_Science_League\n",
    "* https://en.wikipedia.org/wiki/IEEEXtreme (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/TopCoder\n",
    "* https://en.wikipedia.org/wiki/Codeforces\n",
    "* https://en.wikipedia.org/wiki/Facebook_Hacker_Cup\n",
    "* https://en.wikipedia.org/wiki/CodeChef\n",
    "* https://en.wikipedia.org/wiki/CodinGame\n",
    "* https://en.wikipedia.org/wiki/CodeFights\n",
    "* https://en.wikipedia.org/wiki/AI_Challenge\n",
    "* https://en.wikipedia.org/wiki/Google_Code_Jam\n",
    "* https://en.wikipedia.org/wiki/HackerRank\n",
    "* https://en.wikipedia.org/wiki/ICFP_Programming_Contest (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/ACM_International_Collegiate_Programming_Contest (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/Comenius_University\n",
    "* https://en.wikipedia.org/wiki/Online_judge\n",
    "* https://en.wikipedia.org/wiki/SPOJ\n",
    "* https://en.wikipedia.org/wiki/UVa_Online_Judge\n",
    "* PC² (https://en.wikipedia.org/wiki/PC%C2%B2): Programming Contest Control System developed at California State University, Sacramento in support of Computer Programming Contest activities of the ACM, and in particular the ACM International Collegiate Programming Contest.\n",
    "* https://en.wikipedia.org/wiki/Caltech_101"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>Fill a notebook.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "meta"
    ]
   },
   "source": [
    "# Machine learning data sets, benchmarks and competitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most famous datasets:\n",
    "* Iris data set: https://en.wikipedia.org/wiki/Iris_flower_data_set\n",
    "* MNIST database: https://en.wikipedia.org/wiki/MNIST_database\n",
    "\n",
    "List of datasets:\n",
    "* [A large list of datasets from wikipedia](https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research).\n",
    "* [UCI machine learning repository](https://archive.ics.uci.edu/ml/datasets.html): a large collection of datasets for machine learning research.\n",
    "* [Amazon datasets](https://aws.amazon.com/fr/datasets/): Public datasets hosted by Amazon.\n",
    "* [Computer vision datasets](http://www.cvpapers.com/datasets.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "Miscellaneous data:\n",
    "* https://voice.mozilla.org/ : an open source voice data set (see also http://linuxfr.org/news/mozilla-common-voice-vers-une-reconnaissance-vocale-libre)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks and competitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most famous competitions:\n",
    "\n",
    "* [Kaggle](https://www.kaggle.com/) : the most popular competition platform. Contains a large list of [data sets](https://www.kaggle.com/datasets) too. More information on [Wikipedia](https://en.wikipedia.org/wiki/Kaggle).\n",
    "\n",
    "List of competition:\n",
    "\n",
    "* http://www.chalearn.org/challenges.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "**TODO** (from http://www.chalearn.org/challenges.html):\n",
    "* Tunedit: Similar platform more academically oriented (phased out?).\n",
    "* DrivenData: For non-profit challenges.\n",
    "* Codalab: For academic challenges of greater complexity.\n",
    "* Beat: A EU sponsored platform.\n",
    "* Epidemium: challenges in epidemiology.\n",
    "* Pascal challenges: The Pascal network is sponsoring several challenges in Machine learning.\n",
    "* Challenges.gov: Challenges sponsored by the US Government.\n",
    "* Ecole Normale Superieure: Datasets and challenges.\n",
    "* Beaker notebook: Convert back and forth from R/Python/Javascript\n",
    "* Cortana Intelligence: Azure ML platform.\n",
    "* RAMP studio: The Paris-Saclay CDS Rapid Analytics Model Prototyping platform.\n",
    "* Synapse: The platform on which DREAM challenges are organized.\n",
    "\n",
    "Collaborative platforms:\n",
    "* OpenML: share ML reusable frameworks.\n",
    "* MLcomp: compare machine learning programs.\n",
    "* E-lico: data mining portal.\n",
    "* H20: open source predictive analytics platform.\n",
    "* KNIME: Data mining platform.\n",
    "* Quantopian: Financial data simulator + ML tutorials.\n",
    "\n",
    "Crowdsourcing:\n",
    "* Amazon Mechanical Turk: Gets you hire people from all around the world to solve your tasks. Used to label computer vision data.\n",
    "* Crowdflower: Hire people to collect, filter and enhance data.\n",
    "\n",
    "International conferences hosting challenges:\n",
    "* WCCI: World congress on computational intelligence.\n",
    "* ICDAR: International Conference on Document Analysis and Recognition, a bi-annual conference proposing a contest in printed text recognition. Feature extraction/selection is a key component to win such a contest.\n",
    "* ICPR: In conjunction with the International Conference on Pattern Recognition, ICPR 2004, a face recognition contest is being organized.\n",
    "* ICMI: Competitions on multimodal interaction\n",
    "\n",
    "Popular challenges:\n",
    "* NNGC: Neural Network Grand Challenge in time series forecasting.\n",
    "* Netflix: The 1 million dollar Netflix prize, which attracted a lot of attention and broke new grounds for recommender systems.\n",
    "* Robocup: Robots who play soccer, a yearly held contest.\n",
    "* DELVE: A platform developed at University of Torontoto benchmark machine learning algorithms.\n",
    "* CAMDA: Critical Assessment of Microarray Data Analysis, an annual conference on gene expression microarray data analysis. This conference includes a context with emphasis on gene selection, a special case of feature selection.\n",
    "* TREC: Text Retrieval conference, organized every year by NIST. The conference is organized around the result of a competition. Past winners have had to address feature extraction/selection effectively.\n",
    "* CASP: An important competition in protein structure prediction called Critical Assessment of Techniques for Protein Structure Prediction.\n",
    "* ICAPS competitions: Competitions in planning and knowledge engineering\n",
    "* MEDIAEVAL benchmarks: Benchmarking Initiative for Multimedia Evaluation. Data sharing in multimediacommons (with incremental annotations). Uses Amazon web services to allow experimentation in the cloud.\n",
    "* DREAM: Dialogue for Reverse Engineering Assessments and Methods. Challenges in gene network reconstruction.\n",
    "* AVEC: Audio visual Emotion Recognition Challenge and Workshop.\n",
    "* CAFA: Predicting function of biological macromolecules (as well as gene-disease associations).\n",
    "\n",
    "Data resources:\n",
    "* KEEL: Knowledge Extraction based on Evolutionary Learning.\n",
    "* IO Data Science: Datasets of Paris-Saclay University."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "**TODO**: filter the following list...\n",
    "* https://en.wikipedia.org/wiki/Competitive_programming\n",
    "* https://en.wikipedia.org/wiki/American_Computer_Science_League\n",
    "* https://en.wikipedia.org/wiki/IEEEXtreme (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/TopCoder\n",
    "* https://en.wikipedia.org/wiki/Codeforces\n",
    "* https://en.wikipedia.org/wiki/Facebook_Hacker_Cup\n",
    "* https://en.wikipedia.org/wiki/CodeChef\n",
    "* https://en.wikipedia.org/wiki/CodinGame\n",
    "* https://en.wikipedia.org/wiki/CodeFights\n",
    "* https://en.wikipedia.org/wiki/AI_Challenge\n",
    "* https://en.wikipedia.org/wiki/Google_Code_Jam\n",
    "* https://en.wikipedia.org/wiki/HackerRank\n",
    "* https://en.wikipedia.org/wiki/ICFP_Programming_Contest (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/ACM_International_Collegiate_Programming_Contest (a programming competition not especially ML)\n",
    "* https://en.wikipedia.org/wiki/Comenius_University\n",
    "* https://en.wikipedia.org/wiki/Online_judge\n",
    "* https://en.wikipedia.org/wiki/SPOJ\n",
    "* https://en.wikipedia.org/wiki/UVa_Online_Judge\n",
    "* PC² (https://en.wikipedia.org/wiki/PC%C2%B2): Programming Contest Control System developed at California State University, Sacramento in support of Computer Programming Contest activities of the ACM, and in particular the ACM International Collegiate Programming Contest.\n",
    "* https://en.wikipedia.org/wiki/Caltech_101\n",
    "* https://en.wikipedia.org/wiki/LabelMe"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.1442727159416677, 'fraction_numerical': 0.0024928331048236323, 'mean_word_length': 3.491183879093199, 'pattern_counts': {'":': 94, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 59, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4044076', 'n_tokens_mistral': 5015, 'n_tokens_neox': 4867, 'n_words': 1347}