Q:

Need to Merge Avro Files using java application

I am having multiple avro files under a directory which reside on hadoop environment, I need to merge all these files and make it as a single avro file.
example 
/abc->
        x.avro
        y.avro   } => a.avro
        z.avro

The file a.avro will contain contents of all x,y,z files, where x,y,z files having same schema. I need to create a java application. Any help appreciated.
Thanks.

A:

There are few tools provided by the apache avro in order to deal with the avro file operations here. These tools include Merging/Concat tool which merge same schema avro file with non-reserved metadata, catTool to extract samples from an Avro data file, conversion tool which Converts an input file from Avro binary into JSON, recoveryTool which Recovers data from a corrupt Avro Data file etc(Find more on the github url mentioned).
I have extract the code from the same tools mentioned on github, here is the java application that does solve your purpose.
Path inPath  = new Path("C:\\Users\\vaijnathp\\IdeaProjects\\MSExcel\\vaj");
Path outPath = new Path("getDestinationPath") ;
FileSystem fs = FileSystem.get(new Configuration());
FileStatus [] contents contents = fs.listStatus(inPath, new OutputLogFilter());
DataFileWriter<GenericRecord> writer = new DataFileWriter<>(new GenericDatumWriter<>());
Schema schema = null;
String inputCodec = null;
Map<String, byte[]> metadata = new TreeMap<>();
BufferedOutputStream output = new BufferedOutputStream(new BufferedOutputStream(fs.create(outPath)));
for (int i = 0; i < contents.length; i++) {
        FileStatus folderContent = contents[i];
        if (folderContent.isFile() && folderContent.getPath().getName().endsWith(".avro")) {
            InputStream input = new BufferedInputStream(fs.open(folderContent.getPath()));
            DataFileStream<GenericRecord> reader = new DataFileStream<>(input, new GenericDatumReader<GenericRecord>());
            if (schema == null) {
                schema = reader.getSchema();
               //extract metadata for further check.
                extractAvroFileMetadata(writer, metadata, reader);
                inputCodec = reader.getMetaString(DataFileConstants.CODEC);
                if (inputCodec == null) inputCodec = DataFileConstants.NULL_CODEC;
                writer.setCodec(CodecFactory.fromString(inputCodec));
                writer.create(schema, output);
            } else {
                if (!schema.equals(reader.getSchema())) reader.close();
                //compare FileMetadata with previously extracted one
                CompareAvroFileMetadata(metadata, reader, folderContent.getPath().getName());
                String thisCodec = reader.getMetaString(DataFileConstants.CODEC);
                if (thisCodec == null) thisCodec = DataFileConstants.NULL_CODEC;
                if (!inputCodec.equals(thisCodec)) reader.close();
            }
            writer.appendAllFrom(reader, false);
            reader.close();
        }
    }
 writer.close();
}catch (Exception e){
    e.printStackTrace();
}

I hope this code snippet will help you create your java application. Thanks.

