<commit_before>DataVec =========================

[![Join the chat at https://gitter.im/deeplearning4j/deeplearning4j](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/deeplearning4j/deeplearning4j?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.datavec/datavec-api/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.datavec/datavec-api)
[![Javadoc](https://javadoc-emblem.rhcloud.com/doc/org.datavec/datavec-api/badge.svg)](http://deeplearning4j.org/datavecdoc)

DataVec is an Apache2 Licensed open-sourced tool for machine learning ETL (Extract, Transform, Load) operations. The
goal of DataVec is to transform raw data into usable vector formats across machine learning tools.

## Why Would I Use DataVec?

Data handling is sometimes messy, and we believe it should be distinct from high-performance algebra libraries (such
as[nd4j](https://nd4j.org) or [Deeplearning4j](https://deeplearning4j.org)).

DataVec allows a practitioner to take raw data and produce open standard compliant vectorized data (svmLight, etc)
quickly. Current input data types supported out of the box:

* CSV Data
* Raw Text Data (Tweets, Text Documents, etc)
* Image Data


Datavec draws inspiration from a lot of the Hadoop ecosystem tools, and in particular accesses data on disk through the
Hadoop API (like Spark does), which means it's compatible with many records

DataVec also includes sophisticated functionality for feature engineering, data cleaning and data normalization both for
static data and for sequences (time series). Such operations can be executed on [Apache Spark](https://spark.apache.org/) using DataVec-Spark.

## Datavec's architecture : API, transforms and filters, and schema management

Apart from obviously providing readers for classic data formats, DataVec also provides an interface. So if you wanted to<commit_msg>Add to supported record readers.<commit_after>DataVec =========================

[![Join the chat at https://gitter.im/deeplearning4j/deeplearning4j](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/deeplearning4j/deeplearning4j?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.datavec/datavec-api/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.datavec/datavec-api)
[![Javadoc](https://javadoc-emblem.rhcloud.com/doc/org.datavec/datavec-api/badge.svg)](http://deeplearning4j.org/datavecdoc)

DataVec is an Apache2 Licensed open-sourced tool for machine learning ETL (Extract, Transform, Load) operations. The
goal of DataVec is to transform raw data into usable vector formats across machine learning tools.

## Why Would I Use DataVec?

Data handling is sometimes messy, and we believe it should be distinct from high-performance algebra libraries (such
as[nd4j](https://nd4j.org) or [Deeplearning4j](https://deeplearning4j.org)).

DataVec allows a practitioner to take raw data and produce open standard compliant vectorized data (svmLight, etc)
quickly. Current input data types supported out of the box:

* CSV Data
* Raw Text Data (Tweets, Text Documents, etc)
* Image Data
* LibSVM
* SVMLight
* MatLab (MAT) format
* JSON, XML, YAML, XML

Datavec draws inspiration from a lot of the Hadoop ecosystem tools, and in particular accesses data on disk through the
Hadoop API (like Spark does), which means it's compatible with many records

DataVec also includes sophisticated functionality for feature engineering, data cleaning and data normalization both for
static data and for sequences (time series). Such operations can be executed on [Apache Spark](https://spark.apache.org/) using DataVec-Spark.

## Datavec's architecture : API, transforms and filters, and schema management

Apart from obviously providing readers for classic data formats, DataVec also provides an interface. So if you wanted to
{'fraction_non_alphanumeric': 0.1028858218318695, 'fraction_numerical': 0.00602258469259724, 'mean_word_length': 5.6433333333333335, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6041742', 'n_tokens_mistral': 1194, 'n_tokens_neox': 1108, 'n_words': 404}