[mkgmap-dev] splitter improvements

Tue Nov 13 09:31:35 GMT 2012

Thorsten Kukuk wrote
> Correct, I all do is:
> 
> osmconvert planet.osm.pbf -B=osmmaps/scripts/poly/DACH.poly --drop-author
> --drop-version --drop-broken-refs --out-pbf -o=data/osm/DACH.osm.pbf

Ok. First test with r224 crashed with NPE . I did not use the
-drop-broken-refs parm, so I used r225 to test older version.
Both versions passed the point where splitter crashed on your machine, but
both came very close to the limit, so I assume that you were just lucky to
finish with r224.
Some numbers with java version "1.6.0_24":
r225:
Max Heap Space: 2803840K
Max Heap Occupancy: 2614966K

r229:
Max Heap Space: 2799168K
Max Heap Occupancy: 2592023K

r231:
Max Heap Space: 2834944K
Max Heap Occupancy: 2599999K

Ahh, didn't I write that r229 uses less memory? Yes, I did. The explanation
is here: r225 detected fewer problem cases:
r225:
Number of detected problem ways: 133.404
Number of detected problem rels: 32.552

r229 + r231:
Number of detected problem ways: 1.501.122
Number of detected problem rels: 51.845

More interesting are the final numbers of pass 4:
r225:
TreeMap<Long,Integer> node-Writers  : 15.833.747
TreeMap<Long,Integer> way-Writers  : 1.298.226
TreeMap<Long,Integer> rel-Writers  : 44.763

r229 + r231:
TreeMap<Long,Integer> node-Writers  : 28.442.937
TreeMap<Long,Integer> way-Writers  : 2.608.846
TreeMap<Long,Integer> rel-Writers  : 62.263

So, r225 required  almost the same memory to store only *half* of the data.
I was not aware of these big differences. I assume that r229 does a better
job, means, r225 did not find enough problem cases. The results are so
different that it's difficult to say. 
I try to find out using a smaller set of test data. Maybe mkgmap will help
me to detect data that is not used.

Besides that I have an idea how to reduce the memory needs in the critical
pass :-)

Gerd

--
View this message in context: http://gis.19327.n5.nabble.com/splitter-improvements-tp5735230p5735518.html
Sent from the Mkgmap Development mailing list archive at Nabble.com.