[mkgmap-dev] splitter - performance and memory

Tue Jul 21 09:49:56 BST 2009

> It is good to have a more accurate mechanism for determining the
> optimal tile size but, imho, it is better to have a splitter that can
> handle the planet dump on more moderate machines. The advantages of
> more people providing Garmin maps outweigh the little extra handwork
> to get all the tiles rendered and still have reasonable large tile
> sizes. So I would really welcome a version of splitter that uses less
> memory.

I can't see any reason why the Splitter can't be optimised to the point where 
it will handle the planet file on any modest 32bit machine, and that is my 
eventual goal. My first round of changes are just "quick-fixes" really, but 
already they've given good results. With the patch I posted you should be 
able to generate the areas.list file slightly faster and with far less memory 
than before (the actual splitting hasn't changed yet though). Last night 
I also moved the code over to using a different XML parser that has resulted 
in some further gains.

Here's where I currently stand, when splitting the united_kingdom.osm.bz2 
file (165MB) on a Core i7 CPU:

Original splitter.jar - generate areas.list:  4m 24s
Original splitter.jar - splitting stage:        6m 02s
Original splitter.jar - total time taken:    10m 26s

New splitter.jar, generate areas.list:   3m 20s  (~60% memory reduction for 
this stage)
New splitter.jar, splitting stage:         5m 41s  (still requires lots of 
memory)
New splitter.jar, total time taken:       9m 01s

I've got plenty of ideas on ways to further reduce memory and hopefully also 
improve performance, stay tuned.

Chris