[mkgmap-dev] Splitting srtm.osm fails

Thu Jul 28 08:45:41 BST 2011

One possible workaround would be for the splitter to make a pass to determine 
the min and max node IDs (as a 64 bit long), then use the min ID as an adjustment 
to rebase all the node IDs starting from zero. This would still fail if (max 
- min) was greater than 2^31.

A performance improvement would be to allow the min node ID to be specified 
on the command line (assuming the user knew it) or remembered from a previous 
pass.

> On Wed, 27 Jul 2011 19:39:38 +0200, WanMil wrote:
> 
>> It is a regression if r161 works and r179 not so in this case the
>> problem should be tracked down.
>> 
> This is a known limitation of the current splitter. It can handle node
> IDs only up to something less than 2e09 because Java allows only 31
> bit numbers as index for an array.
> 
> Even if this could be solved, you would need huge amounts of memory to
> process data with node IDs starting at 2e09 (like the srtm data),
> because the amount of memory the splitter needs primilary depends on
> the max. node ID (and not on the number of nodes) in the data. If
> there are node IDs larger than about 1.7e09 the splitter needs even
> more memory than a 32 bit JVM can provide.
> 
> Scott has explained this in
> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010598.html and
> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010629.html
> 
> Michael
> 
>>> Hi WanMil,
>>> 
>>> the easiest way for me would be 'splitter-r161'. Do you know, where
>>> to get?
>>> 
>>> BTW, the file with the srtm data is about 4,2 GB and has over 45
>>> million nodes.
>>> 
>>> Josef
>>> 
>>> Am 26.07.2011 21:43, schrieb WanMil:
>>> 
>>>> Hi Josef,
>>>> 
>>>> I have googled for "splitter mkgmap IndexOutOfBoundsException" and
>>>> found a similar problem description:
>>>> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010627.html
>>>> 
>>>> Maybe that helps?
>>>> 
>>>> WanMil
>>>> 
>>>>> No hints?
>>>>> 
>>>>> Am 21.07.2011 08:36, schrieb Josef Latt:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I get srtm data with 'mono Srtm2Osm.exe -bounds1 47.2 5.8 55.2
>>>>>> 15.1 -step 20 -cat 100 20 -large -corrxy 0.0005 0.0005 -o
>>>>>> srtm.osm'
>>>>>> 
>>>>>> Splitting (splitter-r179) the data with 'java -ea -Xmx2048M -jar
>>>>>> /home/jolatt/OSM/splitter/splitter.jar --mapid=21000001
>>>>>> --mixed=yes --max-nodes=5000000 srtm.osm' into tiles I get the
>>>>>> following error:
>>>>>> 
>>>>>> **** WARNING: the default output type has changed to pbf, use
>>>>>> --output=xml for .osm.gz files
>>>>>> 
>>>>>> Elapsed time: 0s   Memory: Current 59MB (1MB used, 58MB free) Max
>>>>>> 1820MB
>>>>>> Time started: Thu Jul 21 08:22:09 CEST 2011
>>>>>> Map is being split for resolution 13:
>>>>>> - area boundaries are aligned to 0x800 map units
>>>>>> - areas are multiples of 0x1000 map units wide and high
>>>>>> Processing srtm.osm
>>>>>> in 1 file
>>>>>> Time: Thu Jul 21 08:22:09 CEST 2011
>>>>>> Exact map coverage is (53.99924039840698,5.800073146820068) to
>>>>>> (55.20007610321045,8.0000638961792)
>>>>>> Trimmed and rounded map coverage is (54.0087890625,6.9873046875)
>>>>>> to
>>>>>> (55.2392578125,8.0419921875)
>>>>>> Splitting nodes into areas containing a maximum of 5.000.000
>>>>>> nodes each...
>>>>>> Area (54.0087890625,6.9873046875) to (55.2392578125,8.0419921875)
>>>>>> contains 2.629 nodes. DONE!
>>>>>> 1 areas:
>>>>>> Area 21000001 covers (0x266800,0x4f800) to (0x274800,0x5b800)
>>>>>> Writing out split osm files Thu Jul 21 08:22:09 CEST 2011
>>>>>> Processing 1 areas in a single pass
>>>>>> (54.0087890625,6.9873046875) to (55.2392578125,8.0419921875)
>>>>>> Starting pass 1 of 1, processing 1 areas (21000001 to 21000001)
>>>>>> Making SparseMultiMap
>>>>>> Making SparseMultiMap
>>>>>> Processing srtm.osm
>>>>>> Exception in thread "main" java.lang.IndexOutOfBoundsException:
>>>>>> Index
>>>>>> (31250001) is greater than or equal to list size (31250001)
>>>>>> at
>>>>>> it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList
>>>>>> .java:258)
>>>>>> at
>>>>>> uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2Sh
>>>>>> ortMapInline.java:128)
>>>>>> at
>>>>>> uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseI
>>>>>> nt2ShortMultiMap.java:81)
>>>>>> at
>>>>>> uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2Sho
>>>>>> rtMultiMap.java:31)
>>>>>> at
>>>>>> uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.j
>>>>>> ava:209)
>>>>>> at
>>>>>> uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor
>>>>>> .java:118)
>>>>>> at
>>>>>> uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:243)
>>>>>> at
>>>>>> uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser
>>>>>> .java:57)
>>>>>> at uk.me.parabola.splitter.Main.processMap(Main.java:412)
>>>>>> at uk.me.parabola.splitter.Main.writeAreas(Main.java:368)
>>>>>> at uk.me.parabola.splitter.Main.split(Main.java:190)
>>>>>> at uk.me.parabola.splitter.Main.start(Main.java:118)
>>>>>> at uk.me.parabola.splitter.Main.main(Main.java:107)
>>>>>> I would try an older version of splitter, but where to download,
>>>>>> or is there another idea?
>>>>>> 
>>>>>> Greetings
>>>>>> Josef Latt
>>>> _______________________________________________
>>>> mkgmap-dev mailing list
>>>> mkgmap-dev at lists.mkgmap.org.uk
>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev