[mkgmap-dev] Patch to reduce memory usage by interning strings.From Scott A Crosby scrosby at cs.rice.edu on Fri Apr 2 04:58:46 BST 2010
On Wed, 31 Mar 2010 22:12:12 +0000 (UTC), Chris Miller <chris_overseas at hotmail.com> writes: > Note that Java's String.intern() method can be pretty slow, so while you'll > save a fair chunk of memory you'll potentially suffer a noticable performance > hit too if you're calling it a lot. By adding a barrier-free caching layer > in front of the String.intern() calls you can gain a reasonable performance > boost in this situation. As an example of how this can be implemented, take > a look at Lucene's SimpleStringInterner which does exactly this: > > http://github.com/apache/lucene/blob/1c5c409241a2b8b9e64dc8c253791b497a66c369/src/java/org/apache/lucene/util/SimpleStringInterner.java > > It's threadsafe in that it guarantees just enough visibility to never generate > invalid results, yet also avoids any blocking. Might be worth benchmarking > something like this against the normal String.intern() with mkgmap. We don't have to get rid of every duplicate string; only most of them, so approximation techniques such as a per-thread or per-parser FuzzyIntern will work fine, and require no concurrent access. Now that I know that String. intern() uses weak references, it seems to be the most trivial way to reduce the memory usage of that tile of planet.osm by at least 3x. Scott
- Previous message: [mkgmap-dev] Patch to reduce memory usage by interning strings.
- Next message: [mkgmap-dev] mkgmap-dev Digest, Vol 21, Issue 2
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list