[mkgmap-dev] Memory limits for mkgmap and splitter

Tue Aug 4 11:02:19 BST 2009

Note that I've started working on overhauling the splitter to try to reduce 
the memory requirements as much as possible without compromising too much 
on performance. I'm also hoping to overcome the limitations on the number 
of tiles (255) and number of areas for each relation (currently 4). My plan 
is to rework some of the algorithms being used, tune the memory requirements 
wherever possible (though Steve's already done a good job here), and switch 
algorithms and page to disk (using eg custom B-Trees, R-Trees, not a database) 
when heap space starts to run out.

So far I've cut the memory requirements for the areas.list file generation 
in half, and hopefully this week I'll check in a further change that slightly 
speeds up the XML parsing and reduces memory requirements a fraction more. 
Further gains beyond that will unfortunately take me a while since it is 
a reasonable amount of work and I don't have a lot of free time to work on 
this. Of course I'll be sure to post to this list when I make any significant 
improvements beyond the above.

As an aside: If you split a file that results in more than 255 tiles, my 
understanding (from looking at the splitter code) is that it will fail 'silently', 
ie the output will be corrupt (nodes/ways/relations end up in the wrong area 
files) without it being obvious that this has occurred. Be careful!

Chris

> Paul Ortyl wrote:
> 
>> Could you explain why you write that
>> "There is a maximum of 255 output files. This should anyway be enough
>> with the current amount of data."
>> ?
> That statement is not true with the current splitter version. When
> splitting North/South America I get 318 tiles.
> 