<commit_before>---
layout: post
title: Nullable schema if read avro fiels by spark
date: 2018-03-26 00:00:00 +00:00
tags: [avro, spark]
---
## Versions
Spark - 2.2

## Problem

If you read avro files by spark you can notice that in the resulted schema all fields are optional. 

For example, if you read avro file with schema to dataframe

{% highlight json %}
{
	"type": "record",
	"name": "user",
	"namespace": "kakfa-avro.test",
	"fields": [
		{
			"name": "id",<commit_msg>fix typo in title<commit_after>---
layout: post
title: Nullable schema if read avro files by spark
date: 2018-03-26 00:00:00 +00:00
tags: [avro, spark]
---
## Versions
Spark - 2.2

## Problem

If you read avro files by spark you can notice that in the resulted schema all fields are optional. 

For example, if you read avro file with schema to dataframe

{% highlight json %}
{
	"type": "record",
	"name": "user",
	"namespace": "kakfa-avro.test",
	"fields": [
		{
			"name": "id",
{'fraction_non_alphanumeric': 0.13778705636743216, 'fraction_numerical': 0.04175365344467641, 'mean_word_length': 3.7475247524752477, 'pattern_counts': {'":': 10, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4811928', 'n_tokens_mistral': 372, 'n_tokens_neox': 333, 'n_words': 130}