[mkgmap-dev] [PATCH]splitter memory usage

Sat Nov 5 23:49:20 GMT 2011

On Thu, 3 Nov 2011 04:25:24 -0700 (PDT), GerdP wrote:

>This patch reduces memory requirement in splitter by ~50% by replacing class
>SparseInt2ShortMultiMap with class Node2AreaMap. 
>It also includes small performance tweaks.

Thank you, Gerd. With this patch applied to the splitter I am able to
process europe.osm.pbf from Geofabrik on a system with a total of only
1.7GB RAM, running Windows XP and Java 1.7 32 bit. Without this patch
the current splitter is completely unusable on that system since it is
not even possible to split a single country (Germany in my case)
without running out of memory. I had to use r161 and OSM data in XML
format, so this patch is a great improvement for me.

>Still not solved by this patch:
>- Memory usage depends on the highest node id in the data.
>- Node id > Integer.MAX_VALUE do not work (not a problem at this moment)
>
>If this small patch is okay for you, I can provide a bigger one that also
>fixes these issues, but touches many more files

Both sounds very interesting to me. If memory usage doesn't depend on
the highest ID in the data but on the number of nodes, it is possible
to split small areas also on small computers. And Node id >
Integer.MAX_VALUE do not work *is* a problem as soon as there is
additional data like contour lines to be added to the OSM data (this
has some advantages compared to an extra layer).

So IMO it is worth considering to have built into the splitter what you
proposed, together with a command line switch allowing to choose
between the "old huge" and the "new small" method.

Michael