[mkgmap-dev] Memory limits for mkgmap and splitterFrom Chris Miller chris.miller at kbcfp.com on Thu Aug 6 11:39:54 BST 2009
Thanks for the info. Given that your run started printing out the areas being split then yes you were very very close to getting the areas.list file. After that's generated, the heap size drops right down and starts growing again on the splitting pass. The smaller --max-nodes value you used would have caused a lot more areas to be created, which will have some impact on the memory required during this first phase though I'm not sure offhand how much. Maybe that's what pushed it over the edge? One thing for sure is that a smaller --max-nodes value will significantly reduce the amount of memory required during the final splitting phase (at the cost of more time due to more areas = more parsing passes). The good news is I just managed to split the entire planet(!) with the following: java -Xmx4000m -jar splitter.jar --max-nodes=1000000 planet-090715.osm It generated 591 areas so made 3 passes over the XML during the splitting stage, and in total the run took 3 hours 32 minutes (overclocked Core i7 with 6GB RAM). Earlier I had tried the split with --max-nodes=1200000 and ran out of memory. I'll add a --max-areas parameter to limit the number of areas processed on each pass. That way you'll be able to trade off performance vs memory of the split stage a bit better using a combination of --max-nodes and --max-areas. Chris > I've ran this splitter against my America extract on a centrino2 > laptop with 4gb ram, below are the results. > > Splitter was working on ~243 million nodes and Java had 3.9 GB heap > space (Xmx3900m). > > Memory usage before I went to bed: 4.3 GB virtual, 3.4 memory, > Splitter was calculating the areas then. > > Hundreds of messages like these followed: > Area (37.4853515625,-123.5302734375) to (38.4521484375,-122.255859375) > contains 439,211 nodes > split horizontally into: > (37.4853515625,-123.5302734375) to (38.4521484375,-122.51953125) > (159,737 nodes) > and (37.4853515625,-122.51953125) to (38.4521484375,-122.255859375) > (279,474 nodes) > Then finally: > > Area (37.4853515625,-122.255859375) to (38.4521484375,-121.728515625) > contains 430,424 nodes > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead > limit > exceeded > at java.lang.Integer.valueOf(Integer.java:601) > at > uk.me.parabola.splitter.AreaSplitter.splitVert(AreaSplitter.java:162) > at > uk.me.parabola.splitter.AreaSplitter.split(AreaSplitter.java:74) > at uk.me.parabola.splitter.Main.calculateAreas(Main.java:175) > at uk.me.parabola.splitter.Main.split(Main.java:96) > at uk.me.parabola.splitter.Main.main(Main.java:79) > I think it was getting close, so a gig more memory would've been > enough perhaps...
- Previous message: [mkgmap-dev] Memory limits for mkgmap and splitter
- Next message: [mkgmap-dev] Memory limits for mkgmap and splitter
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list