Q:

Updating massive amount of rows on database with hibernate

I've been asked to rewrite some batch jobs using Spring Batch, Hibernate and Quartz. The current implementation already uses Hibernate, but there is something wrong in the way they work because they take too much time to complete a task.
This task consists in obtaining items from a XML file and update (or insert, but that does not happen often) the correspondent row in a DB table:
<items>
    <item>
        <id>10005011</id>
        <field_1></field_1> <!--
        <field_2></field_2>
        ...
        <field_n></field_n>
    </item>
    <item>
        <id>23455245</id>
        <field_1></field_1> <!--
        <field_2></field_2>
        ...
        <field_n></field_n>
    </item>

    ...
    <item>
        <id>101000454</id> <!-- about 70000  items-->
        <field_1></field_1> <!--
        <field_2></field_2>
        ...
        <field_n></field_n>
    </item>
</items>

The file is huge, so I've set a chunk size of 1000: the reader takes 1000 items and the writer receives a List of that size to update the table, delegating that to a DAO (ItemDao).
The way it works is the following:

It reads all the items in the table (obtaining a List)
Stores these items in a Map, using their IDs as key.
Loops over the list of Items received as parameter and, one by one, copies all not null fields in the existing items. Hibernate, automagically, updates the modified beans.

The issue I'm facing is each chuck needs more time than the previous one with no apparent reason. Here is my log;
12:33:53,376 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:33:56,927 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:33:59,258 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:34:01,358 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:34:03,145 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:34:31,872 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:35:15,694 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:36:06,211 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:37:02,154 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:38:07,124 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:39:19,519 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:40:34,432 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:41:59,926 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:43:31,951 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:45:12,337 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:46:56,331 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:48:49,726 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:50:48,649 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:52:52,897 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:55:06,056 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:57:28,105 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
12:59:55,983 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:02:40,224 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:05:29,506 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:08:21,031 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:11:18,521 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:14:31,911 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:18:03,994 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:21:43,960 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:25:32,084 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items
13:29:28,366 INFO  [batch.writer.ItemWriter] Updated/inserted 1000 items

Note that the first iterations need a few seconds and as the batch goes on, it takes several minutes... I'm updating about 70000 (seventy thousand) items and last iterations take more than half an hour each.
This is the method in the DAO that is being called:
public void synchronizeItems(List<Item> newItemList,
        Jurisdiction jurisdiction) throws ServiceException {

    Map<Long, Item> ItemMap = new HashMap<Long, Item>();
    List<Item> existingItemList = getAllItems(jurisdiction
            .getJurisdictionId());
    for (Item o : existingItemList) {
        ItemMap.put(o.getProprietorId(), o);
    }

    for (Item newItem : newItemList) {
        updateItem(newItem, jurisdiction, ItemMap);
    }
}

private void updateItem(Item newItem, Jurisdiction jurisdiction,
        Map<Long, Item> ItemMap) throws DAOException {

    Item currItem = ItemMap.get(newItem.getProprietorId());
    if (currItem != null) {
        //just updates currItem, copying all not null attributes from newItem
        copyProperties(currItem, newItem); 
    } else {
        //some times there is a new item
        lspDao.create(newItem);
    }
}

So I have two questions:

Why is each call to synchronizeItems in the loop taking longer that the previous ones?
Is there a better way to update rows?

I was thinking in using a stateless session, and then get all items only once (currently the query is executed once in each loop), so I'll have to call manually to the session.update(Item) manually, something like:
public void batchUpdate(List<T> list) {
    StatelessSession session = sessionFactory.openStatelessSession();
    Transaction tx = session.beginTransaction();
    for(int i=0;i < list.size();i++){
        session.update(list.get(i));
    }
    tx.commit();
}

A:

I do not have experience with Spring Batch, but these guidelines deriving from my work with pure Hibernate may help you:

It is certainly wrong to fetch the whole table from the database. Fetch only those items whose IDs you see in the current XML chunk, using a where item.id in (:ids) clause;
a stateless session cannot be used with persistence management functions of Hibernate (no save, update, merge, and so on---just executeUpdate allowed, which results in an immediate SQL against the database);
the regular, stateful Hibernate Session accumulates all the beans you have involved until either commit or an explicit clear (or in some special circumstances which we shouldn't discuss here).

In a nutshell, this is how the skeleton of the batch update loop should look like:
Session hb = ...;
Transaction tx = ...;
hb.setCacheMode(CacheMode.IGNORE);
hb.setFlushMode(FlushMode.COMMIT);
for (List<Item> chunk : chunks) {
  ... process chunk ...
  hb.flush();
  hb.clear();
}
hb.commit();
tx.close();

Additionally be sure to configure
hibernate.jdbc.batch_size=50

(50 is a good default and it should be kept within the range from 20 to 100). Without this the JDBC Batch API will not be used.

