logo separator

[mkgmap-dev] mkgmap ToDo list

From Gerd Petermann gpetermann_muenchen at hotmail.com on Mon May 5 08:28:43 BST 2014

Hi Lambertus,

attached is a patch that implements the new option:
 --num-tiles       A target value that is used when no split-file is given.
                   Splitting is done so that the given number of tiles is
                   produced. The max-nodes value is ignored if this option is
                   given.
A compiled binary is here:

http://files.mkgmap.org.uk/download/206/splitter.jar

Please let me know if it works as expected.

Gerd
Date: Sun, 4 May 2014 22:20:20 +0200
From: osm at na1400.info
To: mkgmap-dev at lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] mkgmap ToDo list


  
    
  
  
    Great, many thanks. I'm currently trying to create as large as
    (automatically) possible contour tiles and a lot of time is put in
    subsplitting a tile again and again until two subtiles appear. This
    new option would save much time.

    

    On 05/04/2014 10:13 PM, Gerd Petermann
      wrote:

    
    
      
      Hi Lambertus,

        

        yes, I want to code that tomorrow. I prefer to make it "give me
        n tiles", but if that turns out to be 

        difficult I try the simple version. I can think of scenarios
        where it is not possible to create exactly

        n tiles.

        

        Gerd

        

        
          Date: Sun, 4 May 2014 22:08:32 +0200

          From: osm at na1400.info

          To: mkgmap-dev at lists.mkgmap.org.uk

          Subject: Re: [mkgmap-dev] mkgmap ToDo list

          

          Is the 'just give me two tiles' option for splitter on the
          to-do list? I'd really appreciate such a function.

          

          On 05/04/2014 09:36 PM, Gerd
            Petermann wrote:

          
          
            
            Hi Felix,

              

              > would using the precomp-sea parameter for splitter
              improve results? It's not that tiles with loads of sea
              actually failed. 

              

              good question. I've coded that parameter in splitter
              before we did some important changes in mkgmap.

              I think it is needed when you use a polygon file in

              combination with an imput file that doesn't cover the
              polygon. 

              If the polygon covers a costline with a lot of islands,
              but the input file doesn't contain the data,

              splitter might create large tiles covering these "empty"
              areas. Later, in mkgmap, the empty

              areas are filled with the precompiled-sea data, and that
              can cause too large files.

              

              BTW: The quick hack turned out to be useless. It worked
              better in some areas and worse in others.

              I'll look again at this later next week.

              

              Gerd

              

              

              

              
                Date: Sun, 4 May 2014 21:21:59
                +0200

                From: extremecarver at gmail.com

                To: mkgmap-dev at lists.mkgmap.org.uk

                Subject: Re: [mkgmap-dev] mkgmap ToDo list

                

                No - I use splitter like this - with maxnodes depending
                on area - I usually just reduced it after failures to
                compile a tile.:

                java -Xms4000m -Xmx10400m -jar
                c:\openmtbmap\splitter.jar --max-nodes=%maxnodes%
                --max-threads=8 --output=pbf --keep-complete
                --max-areas=1524 --geonames-file=cities5000
                --description=%country% --mapid=%FID%0000
                c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf

                would using the precomp-sea parameter for splitter
                improve results? It's not that tiles with loads of sea
                actually failed. 

                

                On 03.05.2014 09:00,
                  Gerd Petermann wrote:

                
                
                  
                  Hi all,

                    

                    I've coded a quick hack which seems to improve the
                    ratios.

                    On the other hand, I don't see these large
                    differences between

                    smallest and largest img file. 

                    What part of the world should I try?

                    Do you use the precomp-sea parameter in splitter?

                    

                    Gerd

                    

                    
                      Date: Wed, 30 Apr 2014
                      14:59:32 +0200

                      From: extremecarver at gmail.com

                      To: mkgmap-dev at lists.mkgmap.org.uk

                      Subject: Re: [mkgmap-dev] mkgmap ToDo list

                      

                      if that doesn't seriously (more than 30-40%) slow
                      down the splitter, I assume it would be much
                      better...

                      On 30.04.2014
                        14:06, Gerd Petermann wrote:

                      
                      
                        
                          
                          Hi,

                            

                            if I got that right the number of nodes is
                            not 

                            highly correlated to the img size, so the
                            max-nodes

                            value  is not a good estimate.

                            

                            I assume the reason is that nodes which
                            belong to

                            roads produce a lot more bytes

                            in the img file compared to nodes which are
                            parts

                            of shapes or other non-routable ways, not
                            talking

                            about nodes which are simply ignored by the
                            style.

                            

                            So, a possible solution in splitter could be
                            to parse 

                            the ways before reading the nodes and save
                            all nodeids

                            which belong to ways with highway=*.

                            If these nodes are refered by more than one
                            way with highway=*

                            we assume that they will be routing nodes.

                            These special nodes could be counted e.g. 10
                            times to 

                            give a better estimate.

                            

                            Gerd

                            

                            > Date: Wed, 30 Apr 2014 13:36:19
                              +0200

                              > From: osm at na1400.info

                              > To: mkgmap-dev at lists.mkgmap.org.uk

                              > Subject: Re: [mkgmap-dev] mkgmap ToDo
                              list

                              > 

                              > Multithreading the tile rendering for
                              a single map is indeed a bit 

                              > difficult and I gave it up because
                              you need to keep track which image 

                              > id's are already in use. Since I
                              provide multiple maps the work-around 

                              > is running a few scripts parallel,
                              which is also a crude form of 

                              > multithreading.

                              > 

                              > The script language is PHP and it
                              doesn't run on Windows without some 

                              > changes ('/' vs '\' in paths, 'rm
                              -rf', that sort of stuff). Never tried it.

                              > 

                              > To get a better optimum in file size,
                              using the process I described 

                              > earlier, you could start off with a
                              huge --max-nodes setting and then 

                              > 'search' for the highest --max-nodes
                              that works for each specific area.

                              > 

                              > On 30/04/2014 11:49, Felix Hartmann
                              wrote:

                              > > I would love if there was a
                              possibility that you pass the used
                              max-nodes

                              > > value to mkgmap.

                              > > When mkgmap is compiling the
                              maps, then after the .img is created it

                              > > should check

                              > > a) did it crash due to too many
                              max-nodes

                              > > b) for me not important - but
                              for others with very old GPS, etrex 10,

                              > > ---> is tile bigger than X
                              (usually 8) MB.

                              > >

                              > > if a) or b) true, then pass the
                              file back to splitter and split with 60%

                              > > of maxnodes - and compile the
                              resulting .img files again. Should it fail

                              > > again, use 40%, again 25%...
                              Sometimes there are awful tiles, that need

                              > > supersmall max-nodes till they
                              compile, however lately (last 1-2 years)

                              > > I never encountered them
                              anymore. I think that happened rather due
                              to a

                              > > but in splitter/mgkmap that is
                              fixed now.

                              > >

                              > > okay, you could also do this
                              with a script, but it gets rather

                              > > complicated to multithread it
                              (you need to wait till mgkmap finished

                              > > compiling all .img files - and
                              run mkgmap first without address index to

                              > > save time) and do some clever
                              routines on making sure that the map id

                              > > (e.g. 6340????.img) stay
                              correct. Even more complicated to have

                              > > consequent map id...

                              > >

                              > > For europe with a fixed max-node
                              I get tiles from 1.9MB up to 18MB.

                              > > That's factor 9 - so it's
                              huge...

                              > > If I could narrows that down
                              easily to 8-18MB - without getting tiles

                              > > crashing due to too high
                              max-nodes values, that would be sweet.

                              > >

                              > >

                              > >

                              > > As for the scripts - would they
                              run on windows too? - What programming

                              > > language are they in?

                              > >

                              > >

                              > > On 29.04.2014 21:39, osm at na1400.info
                              wrote:

                              > >> Oh, and ofcourse anyone
                              interested can get my scripts, send an
                              email.

                              > >> They'll be on Github someday
                              anyway.

                              > >>

                              > >>

                              > >> On 2014-04-29 20:37, Gerd
                              Petermann wrote:

                              > >>> Hi Lambertus,

                              > >>>

                              > >>> okay, if I got that
                              right you finally get *.img files with a
                              size

                              > >>> near (but below) 8MB, so
                              maybe Henning can use that script, too.

                              > >>>

                              > >>> If you do that for e.g.
                              Germany, how small is tpically the
                              smallest

                              > >>> *.img file ?

                              > >>> Is it probably near 4
                              MB?

                              > >>>

                              > >>> Gerd

                              > >>>

                              > >>>> To: mkgmap-dev at lists.mkgmap.org.uk

                              > >>>> Date: Tue, 29 Apr
                              2014 20:30:27 +0200

                              > >>>> From: osm at na1400.info

                              > >>>> Subject: Re:
                              [mkgmap-dev] mkgmap ToDo list

                              > >>>>

                              > >>>> These are the direct
                              results from Splitter. The format is o5m,
                              both

                              > >>>> input as output.
                              Splitter version is: r321.

                              > >>>>

                              > >>>> For this test I
                              split the original source with
                              --max-nodes=8000000.

                              > >>> Then

                              > >>>> I render the initial
                              tiles, when the result is larger than 8MB
                              it's

                              > >>>> subsplit again with
                              --max-nodes=(8000000-(attempt*100000)).
                              The

                              > >>> initial

                              > >>>> source files are
                              ~70MB (o5m) and after several subsplits
                              the two

                              > >>> *.img

                              > >>>> are < 8MB. During
                              this process --max-nodes has been reduced
                              to e.g.

                              > >>>> 7500000 and the
                              source file is split up in two o5m files
                              of about

                              > >>> 37MB.

                              > >>>>

                              > >>>> I can upload an
                              example source file and it's two subsplit
                              siblings

                              > >>> if

                              > >>>> you want.

                              > >>>>

                              > >>>>

                              > >>>> On 2014-04-29 19:38,
                              GerdP wrote:

                              > >>>> > Hi Lambertus,

                              > >>>> >

                              > >>>> > that's
                              interesting. Are these the img file sizes
                              or the osm file

                              > >>> sizes?

                              > >>>> >

                              > >>>> > Gerd

                              > >>>> >

                              > >>>> >

                              > >>>> > Lambertus wrote

                              > >>>> >>
                              Unfortunately I cannot confirm that. Below
                              is a bit of logging

                              > >>> from my

                              > >>>> >> script:

                              > >>>> >> Original:
                              97000020 (70551453), New: 0 (35684445),
                              New: 1

                              > >>> (36852845)

                              > >>>> >> Original:
                              97000001 (74621042), New: 0 (37522992),
                              New: 1

                              > >>> (37222739)

                              > >>>> >> Original:
                              97000002 (73391358), New: 0 (37679505),
                              New: 1

                              > >>> (38098627)

                              > >>>> >> Original:
                              97000003 (77862567), New: 0 (39075311),
                              New: 1

                              > >>> (39261197)

                              > >>>> >>

                              > >>>> >> The
                              original files above contain contour data,
                              the filesize is

                              > >>> between

                              > >>>> >> brackets.
                              As you can see both resulting file are
                              approximately

                              > >>> the

                              > >>>> >> same

                              > >>>> >> size.

                              > >>>> >>

                              > >>>> >> On
                              2014-04-29 15:39, Gerd Petermann wrote:

                              > >>>> >>> Hi
                              Lambertus,

                              > >>>> >>>

                              > >>>> >>> and I
                              guess that even after this optimization
                              you will

                              > >>>> >>> see a
                              factor 3 or higher between the largest
                              tile and the

                              > >>> smallest.

                              > >>>> >>> Can you
                              confirm that?

                              > >>>> >>>

                              > >>>> >>> Gerd

                              > >>>> >>>

                              > >>>> >>>>
                              Date: Tue, 29 Apr 2014 15:32:38 +0200

                              > >>>> >>>>
                              From:

                              > >>>> >

                              > >>>> >> osm@

                              > >>>> >

                              > >>>> >>>> To:

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>>
                              Subject: Re: [mkgmap-dev] mkgmap ToDo list

                              > >>>> >>>>

                              > >>>> >>>>
                              Num-tiles=x would indeed be better for
                              this specific need.

                              > >>>> >>>>

                              > >>>> >>>> It
                              is my experience that it regularly takes
                              multiple calls to

                              > >>>> >>>
                              Splitter

                              > >>>> >>>> to
                              get 2+ sub-tiles when you reduce the
                              max-nodes by 100k for

                              > >>> each

                              > >>>> >>>>
                              sub-split attempt. This is what I
                              currently do to get an

                              > >>> optimum in

                              > >>>> >>>>
                              tile-size vs total number of tiles.

                              > >>>> >>>>

                              > >>>> >>>>

                              > >>>> >>>>

                              > >>>> >>>> On
                              29/04/2014 15:09, Gerd Petermann wrote:

                              > >>>> >>>>
                              > Hi Lambertus,

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              > that sounds like a possible change in
                              splitter:

                              > >>>> >>>>
                              > Instead of specifying max-nodes you
                              may specify --num-tiles=x

                              > >>>> >>>>
                              > and splitter will try to find a split
                              that produces excactly

                              > >>> x

                              > >>>> >>> tiles

                              > >>>> >>>>
                              > which are not too narrow and have a
                              node number which is not

                              > >>>> >>>>
                              > too far from the average (but still
                              aligned to a multiple of

                              > >>> map

                              > >>>> >>> units

                              > >>>> >>>>
                              > as now).

                              > >>>> >>>>
                              > So, for your script that means you
                              don't have to find the

                              > >>>> >>>
                              max-nodes

                              > >>>> >>>>
                              > value.

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              > I'll think about this again...

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              > Gerd

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              > > Date: Tue, 29 Apr 2014 14:59:36
                              +0200

                              > >>>> >>>>
                              > > From:

                              > >>>> >

                              > >>>> >> osm@

                              > >>>> >

                              > >>>> >>>>
                              > > To:

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>>
                              > > Subject: Re: [mkgmap-dev] mkgmap
                              ToDo list

                              > >>>> >>>>
                              > >

                              > >>>> >>>>
                              > > While this possibly can be
                              solved in Splitter or Mkgmap, it

                              > >>>> >>> could
                              also

                              > >>>> >>>>
                              > > be solved by your build-script
                              when you add a maximum tile

                              > >>> size

                              > >>>> >>> check

                              > >>>> >>>>
                              > > and re-split (with a lower
                              number of max-nodes) until you

                              > >>> get

                              > >>>> >>> two or

                              > >>>> >>>>
                              > > more sub-tiles. Granted, this
                              adds complexity to the script

                              > >>> but

                              > >>>> >>> it
                              works

                              > >>>> >>>>
                              > > well for me.

                              > >>>> >>>>
                              > >

                              > >>>> >>>>
                              > > On 25/04/2014 21:54, Henning
                              Scholland wrote:

                              > >>>> >>>>
                              > > > Hi Gerd,

                              > >>>> >>>>
                              > > >

                              > >>>> >>>>
                              > > > I would like to have
                              img-tiles which have globally nearly

                              > >>> the

                              > >>>> >>> same

                              > >>>> >>>>
                              > > > filesize, so that they use
                              the space of devices like

                              > >>> eTrex 10.

                              > >>>> >>>>
                              > > >

                              > >>>> >>>>
                              > > > With my actual map I use
                              globally the same value for

                              > >>>> >>>
                              max-nodes. But the

                              > >>>> >>>>
                              > > > size of the img-tiles
                              differ more then factor 2. Eg. a

                              > >>> tile in

                              > >>>> >>> Germany

                              > >>>> >>>>
                              > > > is between 2 and 5 mb where
                              a tile in China is about 10

                              > >>> mb. If

                              > >>>> >>> I
                              remove

                              > >>>> >>>>
                              > > > details, this difference
                              will increase, because in

                              > >>> Germany

                              > >>>> >>> more
                              objects

                              > >>>> >>>>
                              > > > will be removed from the
                              img-tile then in China.

                              > >>>> >>>>
                              > > >

                              > >>>> >>>>
                              > > > Henning

                              > >>>> >>>>
                              > > >

                              > >>>> >>>>
                              > > >
                              _______________________________________________

                              > >>>> >>>>
                              > > > mkgmap-dev mailing list

                              > >>>> >>>>
                              > > >

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>>
                              > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >>>>
                              > >

                              > >>>> >>>>
                              > >
                              _______________________________________________

                              > >>>> >>>>
                              > > mkgmap-dev mailing list

                              > >>>> >>>>
                              > >

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>>
                              > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              >

                              > >>>> >>>>
                              >
                              _______________________________________________

                              > >>>> >>>>
                              > mkgmap-dev mailing list

                              > >>>> >>>>
                              >

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>>
                              > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >>>>
                              >

                              > >>>> >>>>

                              > >>>> >>>>
                              _______________________________________________

                              > >>>> >>>>
                              mkgmap-dev mailing list

                              > >>>> >>>>

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >>>

                              > >>>> >>>
                              _______________________________________________

                              > >>>> >>>
                              mkgmap-dev mailing list

                              > >>>> >>>

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >>
                              _______________________________________________

                              > >>>> >> mkgmap-dev
                              mailing list

                              > >>>> >

                              > >>>> >> mkgmap-dev at .org

                              > >>>> >

                              > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>> >

                              > >>>> >

                              > >>>> >

                              > >>>> >

                              > >>>> >

                              > >>>> > --

                              > >>>> > View this
                              message in context:

                              > >>>> >

                              > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html

                              > >>>> > Sent from the
                              Mkgmap Development mailing list archive at

                              > >>> Nabble.com.

                              > >>>> >
                              _______________________________________________

                              > >>>> > mkgmap-dev
                              mailing list

                              > >>>> > mkgmap-dev at lists.mkgmap.org.uk

                              > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>>
                              _______________________________________________

                              > >>>> mkgmap-dev mailing
                              list

                              > >>>> mkgmap-dev at lists.mkgmap.org.uk

                              > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>>

                              > >>>
                              _______________________________________________

                              > >>> mkgmap-dev mailing list

                              > >>> mkgmap-dev at lists.mkgmap.org.uk

                              > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >>
                              _______________________________________________

                              > >> mkgmap-dev mailing list

                              > >> mkgmap-dev at lists.mkgmap.org.uk

                              > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                              > >

                              > 

                              >
                              _______________________________________________

                              > mkgmap-dev mailing list

                              > mkgmap-dev at lists.mkgmap.org.uk

                              > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                            
                          
                        
                        

                        
                        

                        _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
                      
                      

                      -- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org
                      

                      _______________________________________________
                      mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk
                      http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
                  
                  

                  
                  

                  _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
                
                

                -- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org
                

                _______________________________________________
                mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk
                http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
            
            

            
            

            _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
          
          

          

          _______________________________________________
          mkgmap-dev mailing list
          mkgmap-dev at lists.mkgmap.org.uk
          http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
      
      

      
      

      _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
    
    

  


_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140505/35ec71bb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: num-tiles-v1.patch
Type: application/octet-stream
Size: 9235 bytes
Desc: not available
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140505/35ec71bb/attachment-0001.obj>


More information about the mkgmap-dev mailing list