This is a follow-up of my question here:I am currently working on an application for various statistics. One task is to analyse a good amount of sentences for their word counts.The specifications are:sentences are read from SQLiteDatabase (up to 20k with an average of about 15 words)transformation: split by whitespaces (to get the words of the sentences)transformation: toLowerCase (to minimize variations of words)transformation: replace [^a-zA-Z] (for the same reason as above)get word + count for the first x (not sure yet, maybe 10-15) most common wordspreserve a flag if the messages was sent/receivedWhat I'm looking for:improvements to make the code run fasteralternative approaches for this task(general hints to improve the task)Current version with the suggested improvements made//fieldsprivate static final CharMatcher pat_rep = CharMatcher.inRange('A', 'Z').or(CharMatcher.inRange('a', 'z'))        .precomputed();private static final Pattern pat_split = Pattern.compile(\\s);private HashMultiset<String> sent = HashMultiset.create();private HashMultiset<String> rcvd = HashMultiset.create();private Cursor c1;private Cursor c2;//start    c1 = db.rawQuery(select lower(DATA) as SENTENCE, SENT from MESSAGELIST, null);    while (c1.moveToNext()) {        String[] words = pat_split.split(c1.getString(c1.getColumnIndex(SENTENCE)));        int from_me = c1.getInt(c1.getColumnIndex(key_from_me));        for (String in : words) {            in = pat_rep.retainFrom(in);            if (!in.equals()) {                if (from_me == 1) {                    sent.add(in);                } else {                    rcvd.add(in);                }            }        }    }    db.execSQL(create temp table if not exists WORDS (WORD varchar, SENT integer, CNT integer));    SQLiteStatement ins = db.compileStatement(insert into WORDS values (?, ?, ?));    db.beginTransaction();    Iterator<Entry<String>> i = sent.entrySet().iterator();    Entry<String> e = null;    while (i.hasNext()) {        e = i.next();        ins.bindString(1, e.getElement());        ins.bindLong(2, 1);        ins.bindLong(3, e.getCount());        ins.executeInsert();    }    i = rcvd.entrySet().iterator();    while (i.hasNext()) {        e = i.next();        ins.bindString(1, e.getElement());        ins.bindLong(2, 0);        ins.bindLong(3, e.getCount());        ins.executeInsert();    }    db.setTransactionSuccessful();    db.endTransaction();    c1 = db.rawQuery(select WORD, CNT from WORDS where SENT=0 group by WORD order by CNT desc limit 10, null);    c2 = db.rawQuery(select WORD, CNT from WORDS where SENT=1 group by WORD order by CNT desc limit 10, null);
Creating a fast Android dictionary (word counts)
java;performance;android;dictionary;sqlite
Just some ideas.Maybe you could use batching?You may be also able to save some time by iterating over sent.entrySet() instead of looking up the count separately.Split on [^a-zA-Z] as you later throw non-letter away anyway.Can't you use JDK5 loops likefor (String in : send) {...}? I guess, clearBindings is unnecessary as you always overwrite everything.Make all fields private. Always (unless you have a very good reason not to). AT least I hope that pat_rep etc. are fields.Split your method. Shorter methods are easier to read and to optimize.