[mkgmap-dev] Problems with latest splitter versions

Wed Mar 2 23:36:10 GMT 2011

Hi,

I am getting the splitter from svn, compiling and running it unter
Windows XP SP3 using JDK 1.6.0_25. There are 2 problems if I am using a
splitter version newer than r161 (tryed until r167 so far).

1.) germany.osm.pbf from geofabrik cannot be split
--------------------------------------------------

Invoking the splitter with

java.exe -Xms256m -Xmx1536m -server -jar splitter.jar --resolution=14 --overlap=4000 --max-nodes=1200000 --max-areas=50 --geonames-file=cities1000.zip germany.osm.pbf

The data is split into 76 areas, so there are 2 passes to go. First,
everything looks fine, but then, at the and of the first pass, the
splitter "falls asleep", the CPU load for the corresponding JVM drops
to 0% and there ist no further progress (except the memory stats being
printed periodically). The output looks like this (cut a lot of lines):

cache=
description=
geonames-file=cities1000.zip
legacy-mode=false
mapid=63240001
max-areas=50
max-nodes=1200000
max-threads=1 (auto)
mixed=false
no-trim=false
output-dir=
overlap=4000
resolution=14
split-file=
status-freq=120
write-kml=
Elapsed time: 0s   Memory: Current 247MB (1MB used, 246MB free) Max 1484MB
Time started: Wed Mar 02 17:11:11 CET 2011
Map is being split for resolution 14:
 - area boundaries are aligned to 0x400 map units
 - areas are multiples of 0x800 map units wide and high
Processing germany.osm.pbf
Bounding box 5.864417 47.26543 15.050779999000001 55.14777
10.000.000 nodes processed... id=283576079
20.000.000 nodes processed... id=355168786
30.000.000 nodes processed... id=495365986
40.000.000 nodes processed... id=699656373
50.000.000 nodes processed... id=998291790
60.000.000 nodes processed... id=1127935431
1.000.000 ways processed... id=25049151
2.000.000 ways processed... id=28822644
Elapsed time: 1m 59s   Memory: Current 247MB (20MB used, 227MB free) Max 1484MB
3.000.000 ways processed... id=34073254
4.000.000 ways processed... id=40593158
5.000.000 ways processed... id=49931588
6.000.000 ways processed... id=61188933
7.000.000 ways processed... id=83211109
8.000.000 ways processed... id=93106763
9.000.000 ways processed... id=101081600
100.000 ways processed... id=1124462
in 1 file
Time: Wed Mar 02 17:14:00 CET 2011
Exact map coverage is (47.26541519165039,5.864403247833252) to (55.147762298583984,15.05077600479126)
Trimmed and rounded map coverage is (47.28515625,5.8447265625) to (55.1513671875,15.029296875)
Splitting nodes into areas containing a maximum of 1.200.000 nodes each...
Area (47.5048828125,7.470703125) to (47.9443359375,8.7451171875) contains 663.137 nodes. DONE!
[...]
Area (53.8330078125,11.77734375) to (54.931640625,14.2822265625) contains 996.051 nodes. DONE!
76 areas:
Area 63240001 covers (0x246800,0x4b000) to (0x24d800,0x4f800) DE-Dusseldorf
[...]
Area 63240076 covers (0x223000,0x63800) to (0x228800,0x70000) DE-Reutlingen
Writing out split osm files Wed Mar 02 17:14:03 CET 2011
Processing 76 areas in 2 passes, 38 areas at a time
Starting pass 1 of 2, processing 38 areas (63240001 to 63240038)
Making SparseMultiMap
Making SparseMultiMap
Processing germany.osm.pbf
Bounding box 5.864417 47.26543 15.050779999000001 55.14777
Making SparseMultiMap
Making SparseMultiMap
Making SparseMultiMap
MAP occupancy: 832878
MAP occupancy: 158237
MAP occupancy: 8339
MAP occupancy: 546
MAP occupancy: 1652236
MAP occupancy: 326019
MAP occupancy: 19669
MAP occupancy: 2076
MAP occupancy: 2474023
MAP occupancy: 491531
MAP occupancy: 31585
MAP occupancy: 2861
MAP occupancy: 3272326
MAP occupancy: 668457
MAP occupancy: 53049
MAP occupancy: 6168
Elapsed time: 3m 59s   Memory: Current 267MB (121MB used, 146MB free) Max 1484MB
MAP occupancy: 4072409
MAP occupancy: 847909
MAP occupancy: 73212
MAP occupancy: 6470
MAP occupancy: 4870695
MAP occupancy: 1026785
MAP occupancy: 94770
MAP occupancy: 7750
MAP occupancy: 5673776
MAP occupancy: 1203597
MAP occupancy: 111531
MAP occupancy: 11096
10.000.000 nodes processed... id=283576079
[...]
60.000.000 nodes processed... id=1127935431
MAP occupancy: 37091591
MAP occupancy: 8145968
MAP occupancy: 704292
MAP occupancy: 58149
MAP occupancy: 37918126
MAP occupancy: 8301600
MAP occupancy: 722028
MAP occupancy: 58246
MAP occupancy: 38711472
MAP occupancy: 8485566
MAP occupancy: 744601
MAP occupancy: 58361
Writing ways Wed Mar 02 17:26:02 CET 2011
Making SparseMultiMap
Making SparseMultiMap
Making SparseMultiMap
1.000.000 ways processed... id=25049151
MAP occupancy: 798645
MAP occupancy: 184591
MAP occupancy: 15271
MAP occupancy: 1493
Elapsed time: 15m 59s   Memory: Current 1484MB (705MB used, 779MB free) Max 1484MB
2.000.000 ways processed... id=28822644
[...]
7.000.000 ways processed... id=83211109
Elapsed time: 20m 2s   Memory: Current 1484MB (928MB used, 556MB free) Max 1484MB
MAP occupancy: 4795349
MAP occupancy: 1103602
MAP occupancy: 94312
MAP occupancy: 6736
MAP occupancy: 1
8.000.000 ways processed... id=93106763
9.000.000 ways processed... id=101081600
MAP occupancy: 5597978
MAP occupancy: 1283033
MAP occupancy: 110757
MAP occupancy: 8231
MAP occupancy: 1
Writing relations Wed Mar 02 17:32:01 CET 2011
100.000 ways processed... id=1124462
coords occupancy
MAP occupancy: 39240504
MAP occupancy: 8605635
MAP occupancy: 752197
MAP occupancy: 58361
ways occupancy
MAP occupancy: 5634593
MAP occupancy: 1292952
MAP occupancy: 111321
MAP occupancy: 8235
MAP occupancy: 1
Elapsed time: 22m 2s   Memory: Current 1484MB (1144MB used, 340MB free) Max 1484MB
Elapsed time: 24m 2s   Memory: Current 1484MB (1144MB used, 340MB free) Max 1484MB
Elapsed time: 26m 2s   Memory: Current 1484MB (1144MB used, 340MB free) Max 1484MB
Elapsed time: 28m 2s   Memory: Current 1484MB (1144MB used, 340MB free) Max 1484MB
Elapsed time: 30m 2s   Memory: Current 1484MB (1144MB used, 340MB free) Max 1484MB

At this point, nothing more happens, no second pass is done.

2.) File containing contour data cannot be processed
----------------------------------------------------

When generating my maps, I usually add data to draw contour lines. The
file containing that data is available from
http://osm.arndnet.de/contourdata.osm.gz (not my site!). It is in OSM
format 0.5.

When buildig a map, I am  combining this data with the OSM data (please
note that the contour data file contains node IDs starting at 2^31, so
they collide with the IDs from "real" OSM data if used unchanged). But
even when I am trying to split this file without additional OSM data,
the same as above happens and the splitter stops at the end of the
first and only pass:

java.exe -Xms256m -Xmx1536m -server -jar splitter.jar --resolution=14 --overlap=4000 --max-nodes=1200000 --max-areas=50 --mixed --geonames-file=cities1000.zip contourdata.osm.gz

cache=
description=
geonames-file=cities1000.zip
legacy-mode=false
mapid=63240001
max-areas=50
max-nodes=1200000
max-threads=1 (auto)
mixed=true
no-trim=false
output-dir=
overlap=4000
resolution=14
split-file=
status-freq=120
write-kml=
Elapsed time: 0s   Memory: Current 247MB (1MB used, 246MB free) Max 1484MB
Time started: Wed Mar 02 17:43:31 CET 2011
Map is being split for resolution 14:
 - area boundaries are aligned to 0x400 map units
 - areas are multiples of 0x800 map units wide and high
Processing contourdata.osm.gz
2.500.000 nodes processed...
5.000.000 nodes processed...
7.500.000 nodes processed...
10.000.000 nodes processed...
12.500.000 nodes processed...
15.000.000 nodes processed...
17.500.000 nodes processed...
in 1 file
Time: Wed Mar 02 17:45:08 CET 2011
Exact map coverage is (47.2760796546936,5.877492427825928) to (55.02375841140747,14.999899864196777)
Trimmed and rounded map coverage is (47.28515625,5.86669921875) to (55.0634765625,15.00732421875)
Splitting nodes into areas containing a maximum of 1.200.000 nodes each...
Area (47.5048828125,6.7236328125) to (49.1748046875,8.525390625) contains 958.841 nodes. DONE!
[...]
Area (52.822265625,10.17333984375) to (54.7119140625,14.47998046875) contains 885.962 nodes. DONE!
22 areas:
Area 63240001 covers (0x23c000,0x54c00) to (0x247800,0x5ec00) DE-Koblenz
[...]
Area 63240022 covers (0x21c800,0x61000) to (0x22f800,0x6e800) DE-Stuttgart
Writing out split osm files Wed Mar 02 17:45:10 CET 2011
Processing 22 areas in a single pass
Starting pass 1 of 1, processing 22 areas (63240001 to 63240022)
Making SparseMultiMap
Making SparseMultiMap
Processing contourdata.osm.gz
Making SparseMultiMap
Writing ways Wed Mar 02 17:45:14 CET 2011
Making SparseMultiMap
Making SparseMultiMap
Making SparseMultiMap
Making SparseMultiMap
Making SparseMultiMap
Elapsed time: 1m 59s   Memory: Current 1484MB (1090MB used, 394MB free) Max 1484MB
MAP occupancy: 783578
MAP occupancy: 190519
MAP occupancy: 25903
MAP occupancy: 1545344
MAP occupancy: 394559
MAP occupancy: 60097
MAP occupancy: 2398068
MAP occupancy: 533001
MAP occupancy: 68931
2.500.000 nodes processed...
MAP occupancy: 3315227
MAP occupancy: 615842
MAP occupancy: 68931
MAP occupancy: 4195362
MAP occupancy: 728752
MAP occupancy: 75886
5.000.000 nodes processed...
[...]
[...]
Elapsed time: 7m 59s   Memory: Current 1484MB (1197MB used, 287MB free) Max 1484MB
MAP occupancy: 16438198
MAP occupancy: 3986436
MAP occupancy: 575366
MAP occupancy: 17300217
MAP occupancy: 4121087
MAP occupancy: 578696
17.500.000 nodes processed...
coords occupancy
MAP occupancy: 17896316
MAP occupancy: 4149555
MAP occupancy: 578696
ways occupancy
MAP occupancy: 180407
MAP occupancy: 43178
MAP occupancy: 6700
MAP occupancy: 21
MAP occupancy: 1
Elapsed time: 9m 59s   Memory: Current 1484MB (1013MB used, 471MB free) Max 1484MB
Elapsed time: 11m 59s   Memory: Current 1484MB (1014MB used, 470MB free) Max 1484MB
Elapsed time: 13m 59s   Memory: Current 1484MB (1014MB used, 470MB free) Max 1484MB
Elapsed time: 15m 59s   Memory: Current 1484MB (1014MB used, 470MB free) Max 1484MB
***** Full GC *****
Elapsed time: 18m 1s   Memory: Current 1484MB (923MB used, 561MB free) Max 1484MB
Elapsed time: 20m 1s   Memory: Current 1484MB (924MB used, 560MB free) Max 1484MB
Elapsed time: 22m 1s   Memory: Current 1484MB (924MB used, 560MB free) Max 1484MB

and so on.

I've then converted the data to OSM format 0.6 and sorted it (using
--migrate and --sort-0.6 for osmosis) and stored it as .osm.bz2 and
.osm.pbf. When I am trying to split these files, I am getting an
exception:

java.exe -Xms256m -Xmx1536m -server -jar splitter.jar --resolution=14 --overlap=4000 --max-nodes=1200000 --max-areas=50 --geonames-file=cities1000.zip contourdata06.osm.bz2

cache=
description=
geonames-file=cities1000.zip
legacy-mode=false
mapid=63240001
max-areas=50
max-nodes=1200000
max-threads=1 (auto)
mixed=false
no-trim=false
output-dir=
overlap=4000
resolution=14
split-file=
status-freq=120
write-kml=
Elapsed time: 0s   Memory: Current 247MB (1MB used, 246MB free) Max
1484MB
Time started: Wed Mar 02 19:28:48 CET 2011
Map is being split for resolution 14:
 - area boundaries are aligned to 0x400 map units
 - areas are multiples of 0x800 map units wide and high
Processing contourdata06.osm.bz2
2.500.000 nodes processed...
5.000.000 nodes processed...
7.500.000 nodes processed...
Elapsed time: 2m 0s   Memory: Current 247MB (81MB used, 166MB free) Max 1484MB
10.000.000 nodes processed...
12.500.000 nodes processed...
15.000.000 nodes processed...
Elapsed time: 4m 0s   Memory: Current 247MB (61MB used, 186MB free) Max 1484MB
17.500.000 nodes processed...
in 1 file
Time: Wed Mar 02 19:34:21 CET 2011
Exact map coverage is (47.2760796546936,5.877492427825928) to (55.02375841140747,14.999899864196777)
Trimmed and rounded map coverage is (47.28515625,5.86669921875) to (55.0634765625,15.00732421875)
Splitting nodes into areas containing a maximum of 1.200.000 nodes each...
Area (47.5048828125,6.7236328125) to (49.1748046875,8.525390625) contains 958.841 nodes. DONE!
[...]
Area (52.822265625,10.17333984375) to (54.7119140625,14.47998046875) contains 885.962 nodes. DONE!
22 areas:
Area 63240001 covers (0x23c000,0x54c00) to (0x247800,0x5ec00) DE-Koblenz
[...]
Area 63240022 covers (0x21c800,0x61000) to (0x22f800,0x6e800) DE-Stuttgart
Writing out split osm files Wed Mar 02 19:34:24 CET 2011
Processing 22 areas in a single pass
Starting pass 1 of 1, processing 22 areas (63240001 to 63240022)
Making SparseMultiMap
Making SparseMultiMap
Processing contourdata06.osm.bz2

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Start index (-30655339) is negative
        at it.unimi.dsi.fastutil.Arrays.ensureFromTo(Arrays.java:45)
        at it.unimi.dsi.fastutil.objects.ObjectArrays.ensureFromTo(ObjectArrays.java:351)
        at it.unimi.dsi.fastutil.objects.ObjectArrays.fill(ObjectArrays.java:313) 
        at it.unimi.dsi.fastutil.objects.ObjectArrayList.size(ObjectArrayList.java:308)
        at uk.me.parabola.splitter.SparseInt2ShortMapInline.resizeTo(SparseInt2ShortMapInline.java:96)
        at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortMapInline.java:123)
        at uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2ShortMultiMap.java:81)
        at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMultiMap.java:31)
        at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:208)
        at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:118)
        at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:243)
        at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:57)
        at uk.me.parabola.splitter.Main.processMap(Main.java:399)
        at uk.me.parabola.splitter.Main.writeAreas(Main.java:355)
        at uk.me.parabola.splitter.Main.split(Main.java:188)
        at uk.me.parabola.splitter.Main.start(Main.java:116)
        at uk.me.parabola.splitter.Main.main(Main.java:105)

Both files (the original file in OSM format 0.5 and the converted
.osm.bz2 in OSM format 0.6) could be split without a problem using
splitter r161.

Any ideas what is going wrong there? Anything I could try? 

Michael