Q:

spring batch multiple sources in multiple subdirectory

I am new to spring batch and going though spring batch and read multipartItemReder , I guess multipartItemReader is not fit for my project . Please through your thoughts and guild be by giving few points . 
I have more then 50 million xml files , like the below directory structure . 
 GOOD
    0
      001/en/1.xml
      001/jp/1.xml
      002/en/2.xml
      003/en/3.xml
      004/jp/4.xml
       .... 
       ....
      999/jp/1.xml  
    1000
      001/en/1.xml
      001/jp/1.xml
      002/en/2.xml
      003/en/3.xml
      004/jp/4.xml
       .... 
       ....
      999/jp/1.xml  
    2000
    3000
    ... 
    .. no limit 
 REMAKE/
    0
      001/en/1.xml
      001/jp/1.xml
      002/en/2.xml
      003/en/3.xml
      004/jp/4.xml
       .... 
       ....
      999/jp/1.xml  
 PROCLAIMED/
 ...
  ...
  ....
  like 100 directories ..  

each source(GOOD,REMAKE, PROCLAIMED ... etc)  has different  xml file formate .
1.I need to create Item processor for each  source . 
2.Each source will be a thread or with commit transaction=1 or thread span based on the number of lang files inside SOURCE////.xml , What is the better option . 
3.I still feel IteamReader is complex implementation. Here each xml file has one record only. please share your comments .
Thanks

A:

Probably the best pratice for this kind of scenario is to use
partitioning; I haven't tried it so I can't help a lot, but I think partitioning is helpful when you have the same type data to manage and not in your case where data is a mix.
Now my 2 cents...
I'll go for parallel steps

Each source managed as a separated thread using split/flow
No need to have commit-interval equals 1; you can use a large value (or a custom CompletionPolicy if you want a fine grained commit) to improve performance
Use a MultiResourceItemReader delegating to a StaxEventItemReader specific for every kind of source
Specialized processor for every kind of object returned by reader
Writer (depends of your needs)

<job id="job1">
  <split id="split1" task-executor="taskExecutor" next="lastStep">
    <flow>
      <step id="GOOD" />
    </flow>
    <flow>
      <step id="REMAKE" />
    </flow>
    <flow>
      <step id="PROCLAIMED" />
    </flow>
  </split>
  <step id="GOOD">
    <tasklet>
      <batch commit-interval="100">
        // Set MultiResourceItemReader and delegate to specialized StaxEventItemReader for GOOD file structure
        // Set specialized processor for GOOD object
        // Set writer (IDK which type)
      </batch>
  </step>
</job>

