[mkgmap-dev] Splitter question

Tue Nov 16 18:12:01 GMT 2010

On Mon, 2010-11-15 at 14:28 -0600, Scott Crosby wrote:
>         Node 986712679 in too many areas. Already in areas 0xd0a0706,
>         trying to
>         add area 0x12
> 
> 
> The tracking tables used by the old version of the splitter can only
> track nodes that occur in at most 4 output files. (To handler borders
> correctly, nodes near a region are included in it.) The warnings
> indicate that nodes weren't written to a file that they should have
> been. That's bad.
> 
> 
> I suggest using a copy of the splitter generated from the
> crosby_integration branch. The code in that branch supports pbf input,
> runs faster, and doesn't have this limitation.
>  http://www.mail-archive.com/mkgmap-dev@lists.mkgmap.org.uk/msg07266.html

This appears to be better indeed.

> It sounds like the limit is the minimum size of a region. Note that
> this is a limitation of the part of the splitter that automatically
> finds splits. If areas.list is manually edited, smaller regions can be
> generated.
> 
> 
> You're hitting that minimum-area-size limit at anything under 10,000
> nodes. The only fixes are either increase the resolution, or, for
> regions containing too many nodes, edit areas.list file and subdivide
> the regions that are too big.
> 
> 
>         max-nodes=10000 , resolution=14 --> 6 files, "only" 120000
>         lines of
>         output in typescript.
>         
>         BTW I also tried decreasing max-nodes without changing the
>         resolution,
>         this is what Felix wrote. That didn't work : the file never
>         got split
>         further.
>         
>         So how do I describe what to do, in documentation for RoadMap
>         users ?
> 
> 
> First question is how many nodes can RoadMap handle?

There are several limits in RoadMap. The one I'm hitting right now (with
areas not sufficiently split) is the number of "polygons" which is
limited to 65535. 

Here's a sample output of buildmap on a file that doesn't hit the limit,
to give you an idea :
pavilion:
{1062} /home/danny/src/roadmap/roadmap.cvs.sf.net/roadmap/build/linux/buildmap_osm -c ../default/All -m maps.gb -i iso-gb-881.osm -o iso-gb-881.rdm
-- Pass 1 : 262313 lines read (0 seconds)
-- Pass 2 : 262313 lines read (3 seconds)
-- Pass 3 : 262313 lines read (0 seconds)
-- loading shape info (from 47261 ways) ...
-- generating squares...
-- sorting squares...
-- sorting points...
-- counting crossings...
-- sorting lines...
-- Pass 4 : 262313 lines read (0 seconds)
-- Splits 39193, ways split 6103, not split 2595
-- sorting streets...
-- retrieving lines and squares...
-- sorting polygons' squares...
-- sorting polygons...
-- sorting polygon lines...
-- writing results to iso-gb-881.rdm
-- saving dictionary...
-- saving 5 attributes...
-- saving 89604 points...
-- saving 57180 lines...
-- Line By Point : 89602 points, 114359 lines
-- saving 2406 streets...
-- building the street search accelerator...
-- saving 0 ranges...
-- saving 9919 polygons...
-- saving 36 squares...
pavilion: {1063} ls -l iso-gb-881.osm maps.gb/iso-gb-881.rdm
-rw-rw-r-- 2 danny danny 8988883 2010-11-13 16:18 iso-gb-881.osm
-rw-rw-r-- 1 danny danny 2328604 2010-11-16 19:06 maps.gb/iso-gb-881.rdm

I just ran it on the OSM file for the Netherlands, with command
  java -jar $SPLITTER --mapid=001 --max-nodes=40000 --resolution=14
$OSMFILE

and got this error :
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at
it.unimi.dsi.fastutil.longs.LongArrays.ensureCapacity(LongArrays.java:107)
        at
it.unimi.dsi.fastutil.longs.LongArrayList.ensureCapacity(LongArrayList.java:202)
        at
it.unimi.dsi.fastutil.longs.LongArrayList.size(LongArrayList.java:271)
        at
uk.me.parabola.splitter.SparseInt2ShortMapInline.resizeTo(SparseInt2ShortMapInline.java:95)
        at
uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortMapInline.java:123)
        at uk.me.parabola.splitter.SparseInt2ShortMultiMap
$Inner.put(SparseInt2ShortMultiMap.java:81)
        at uk.me.parabola.splitter.SparseInt2ShortMultiMap
$Inner.put(SparseInt2ShortMultiMap.java:79)
        at
uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMultiMap.java:31)
        at
uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:208)
        at
uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:118)
        at
uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:243)
        at
uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:57)
        at uk.me.parabola.splitter.Main.processMap(Main.java:336)
        at uk.me.parabola.splitter.Main.writeAreas(Main.java:307)
        at uk.me.parabola.splitter.Main.split(Main.java:152)
        at uk.me.parabola.splitter.Main.start(Main.java:108)
        at uk.me.parabola.splitter.Main.main(Main.java:97)

Earlier runs with max-nodes=10000 and 20000 were going to generate too
many files (> 1000) so I ended up interrupting them after pass 1 but
they didn't crash before that point.

Still looking for good advice on how to document this ;-)

	Danny

-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info