[mkgmap-dev] Memory limits for mkgmap and splitter

Thu Aug 6 11:39:54 BST 2009

Thanks for the info. Given that your run started printing out the areas being 
split then yes you were very very close to getting the areas.list file. After 
that's generated, the heap size drops right down and starts growing again 
on the splitting pass. The smaller --max-nodes value you used would have 
caused a lot more areas to be created, which will have some impact on the 
memory required during this first phase though I'm not sure offhand how much. 
Maybe that's what pushed it over the edge? One thing for sure is that a smaller 
--max-nodes value will significantly reduce the amount of memory required 
during the final splitting phase (at the cost of more time due to more areas 
= more parsing passes).

The good news is I just managed to split the entire planet(!) with the following:

java -Xmx4000m -jar splitter.jar --max-nodes=1000000 planet-090715.osm

It generated 591 areas so made 3 passes over the XML during the splitting 
stage, and in total the run took 3 hours 32 minutes (overclocked Core i7 
with 6GB RAM). Earlier I had tried the split with --max-nodes=1200000 and 
ran out of memory.

I'll add a --max-areas parameter to limit the number of areas processed on 
each pass. That way you'll be able to trade off performance vs memory of 
the split stage a bit better using a combination of --max-nodes and --max-areas.

Chris

> I've ran this splitter against my America extract on a centrino2
> laptop with 4gb ram, below are the results.
> 
> Splitter was working on ~243 million nodes and Java had 3.9 GB heap
> space (Xmx3900m).
> 
> Memory usage before I went to bed: 4.3 GB virtual, 3.4 memory,
> Splitter was calculating the areas then.
> 
> Hundreds of messages like these followed:
> Area (37.4853515625,-123.5302734375) to (38.4521484375,-122.255859375)
> contains 439,211 nodes
> split horizontally into:
> (37.4853515625,-123.5302734375) to (38.4521484375,-122.51953125)
> (159,737 nodes)
> and (37.4853515625,-122.51953125) to (38.4521484375,-122.255859375)
> (279,474 nodes)
> Then finally:
> 
> Area (37.4853515625,-122.255859375) to (38.4521484375,-121.728515625)
> contains 430,424 nodes
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead
> limit
> exceeded
> at java.lang.Integer.valueOf(Integer.java:601)
> at
> uk.me.parabola.splitter.AreaSplitter.splitVert(AreaSplitter.java:162)
> at
> uk.me.parabola.splitter.AreaSplitter.split(AreaSplitter.java:74)
> at uk.me.parabola.splitter.Main.calculateAreas(Main.java:175)
> at uk.me.parabola.splitter.Main.split(Main.java:96)
> at uk.me.parabola.splitter.Main.main(Main.java:79)
> I think it was getting close, so a gig more memory would've been
> enough perhaps...