<commit_before>hive-json-split
===============

A simple UDF to split JSON arrays into Hive arrays.

Building
===============

Check out the code and run 

```
   mvn package
```

to build an uberjar with everything you need. 

Usage
===============

The UDF accepts a single JSON string containing only an array. In the Hive CLI:

```
add jar target/JsonSplit-1.0-SNAPSHOT.jar;
create temporary function json_split as 'com.pythian.hive.udf.JsonSplitUDF';

create table json_example (json string);
load data local inpath 'example.json' into table json_example;

SELECT ex.* FROM json_example LATERAL VIEW explode(json_split(json_example.json)) ex;
```

The json_split function by itself takes a json string like '[1,2,3]' and turns it into an array of tuples: 

```
[
  {
    row_id:1, 
    json_string:'1' 
  },
  { 
    row_id:2, 
    json_string:'2' 
  }, 
  {
    row_id:3, 
    json_string:'3' 
  }
]
```

Then you can use the built-in explode UDTF and LATERAL VIEW to produce one record per element in the JSON array.<commit_msg>Final README changes<commit_after>hive-json-split
===============

A simple UDF to split JSON arrays into Hive arrays.

Building
===============

Check out the code and run 

```
   mvn package
```

to build an uberjar with everything you need. 

Usage
===============

The UDF accepts a single JSON string containing only an array. In the Hive CLI:

```
add jar target/JsonSplit-1.0-SNAPSHOT.jar;
create temporary function json_split as 'com.pythian.hive.udf.JsonSplitUDF';

create table json_example (json string);
load data local inpath 'example.json' into table json_example;

SELECT ex.* FROM json_example LATERAL VIEW explode(json_split(json_example.json)) ex;
```

The json_split function by itself takes a json string like '[1,2,3]' and turns it into an array of tuples: 

```
[
  {
    row_id:1, 
    json_string:'1' 
  },
  { 
    row_id:2, 
    json_string:'2' 
  }, 
  {
    row_id:3, 
    json_string:'3' 
  }
]
```

Then you can use the built-in explode UDTF and LATERAL VIEW to produce one record per element in the JSON array. 

You can access the JSON string for the element with the json_string attribute. Row_id is the position in the array.
{'fraction_non_alphanumeric': 0.12300683371298406, 'fraction_numerical': 0.010022779043280182, 'mean_word_length': 3.7429805615550755, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7161415', 'n_tokens_mistral': 756, 'n_tokens_neox': 704, 'n_words': 261}