logo separator

[mkgmap-dev] mkgmap ToDo list

From Gerd Petermann gpetermann_muenchen at hotmail.com on Mon May 5 05:54:46 BST 2014

Hi Felix,

thanks for the info, I was not aware of any "number of tiles" limit. 
I have a new theory regarding "density of nodes"  and "img file size".

With the styles I use for testing, the largest tiles (img file size) are typically
also large regarding the bbox.

It seems that points in areas with rather low density are likely
to produce more bytes compared to those in very dense areas.
I see two possible reasons:
1) Many points in dense areas are ignored
2) For lines and shapes, the amount of points is not the only
factor that determines the number of needed bytes, it also
matters how big the largest delta values are (that is the lon/lat-delta
between two consecutive points)

Maybe we can find a simple rule to weight nodes in dense areas
lower than those in rarely populated areas.

Gerd

Date: Mon, 5 May 2014 01:24:31 +0200
From: extremecarver at gmail.com
To: mkgmap-dev at lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] mkgmap ToDo list


  
    
  
  
    I just tried asia with and without the precomp-sea option.

    Without I even quicker run into broken tiles (1). Using it gives me
    3 more tiles total (660 overall), and no failing tile.

    However - smallest tile is 970KB while biggest is 15MB (around 19MB
    is around the biggest I see from mkgmap without crash). So that's a
    more than 15 ratio.

    

    Long way to go I would say. Though it only really matters to me for
    continent maps, and Germany/China/Canada/USA/France maps - those are
    likely to push people into 2025 or 4000 tile limit on their devices
    if too many small tiles exist...

    Europe map is anyhow over the tops... (with my style-file - would
    need very reduced style for getting it <4GB).

    

    A range of 5-18MB would already be very nice...

    

    On 03.05.2014 09:00, Gerd Petermann
      wrote:

    
    
      
      Hi all,

        

        I've coded a quick hack which seems to improve the ratios.

        On the other hand, I don't see these large differences between

        smallest and largest img file. 

        What part of the world should I try?

        Do you use the precomp-sea parameter in splitter?

        

        Gerd

        

        
          Date: Wed, 30 Apr 2014 14:59:32 +0200

          From: extremecarver at gmail.com

          To: mkgmap-dev at lists.mkgmap.org.uk

          Subject: Re: [mkgmap-dev] mkgmap ToDo list

          

          if that doesn't seriously (more than 30-40%) slow down the
          splitter, I assume it would be much better...

          On 30.04.2014 14:06, Gerd
            Petermann wrote:

          
          
            
              
              Hi,

                

                if I got that right the number of nodes is not 

                highly correlated to the img size, so the max-nodes

                value  is not a good estimate.

                

                I assume the reason is that nodes which belong to

                roads produce a lot more bytes

                in the img file compared to nodes which are parts

                of shapes or other non-routable ways, not talking

                about nodes which are simply ignored by the style.

                

                So, a possible solution in splitter could be to parse 

                the ways before reading the nodes and save all nodeids

                which belong to ways with highway=*.

                If these nodes are refered by more than one way with
                highway=*

                we assume that they will be routing nodes.

                These special nodes could be counted e.g. 10 times to 

                give a better estimate.

                

                Gerd

                

                > Date: Wed, 30 Apr 2014 13:36:19 +0200

                  > From: osm at na1400.info

                  > To: mkgmap-dev at lists.mkgmap.org.uk

                  > Subject: Re: [mkgmap-dev] mkgmap ToDo list

                  > 

                  > Multithreading the tile rendering for a single
                  map is indeed a bit 

                  > difficult and I gave it up because you need to
                  keep track which image 

                  > id's are already in use. Since I provide multiple
                  maps the work-around 

                  > is running a few scripts parallel, which is also
                  a crude form of 

                  > multithreading.

                  > 

                  > The script language is PHP and it doesn't run on
                  Windows without some 

                  > changes ('/' vs '\' in paths, 'rm -rf', that sort
                  of stuff). Never tried it.

                  > 

                  > To get a better optimum in file size, using the
                  process I described 

                  > earlier, you could start off with a huge
                  --max-nodes setting and then 

                  > 'search' for the highest --max-nodes that works
                  for each specific area.

                  > 

                  > On 30/04/2014 11:49, Felix Hartmann wrote:

                  > > I would love if there was a possibility that
                  you pass the used max-nodes

                  > > value to mkgmap.

                  > > When mkgmap is compiling the maps, then
                  after the .img is created it

                  > > should check

                  > > a) did it crash due to too many max-nodes

                  > > b) for me not important - but for others
                  with very old GPS, etrex 10,

                  > > ---> is tile bigger than X (usually 8)
                  MB.

                  > >

                  > > if a) or b) true, then pass the file back to
                  splitter and split with 60%

                  > > of maxnodes - and compile the resulting .img
                  files again. Should it fail

                  > > again, use 40%, again 25%... Sometimes there
                  are awful tiles, that need

                  > > supersmall max-nodes till they compile,
                  however lately (last 1-2 years)

                  > > I never encountered them anymore. I think
                  that happened rather due to a

                  > > but in splitter/mgkmap that is fixed now.

                  > >

                  > > okay, you could also do this with a script,
                  but it gets rather

                  > > complicated to multithread it (you need to
                  wait till mgkmap finished

                  > > compiling all .img files - and run mkgmap
                  first without address index to

                  > > save time) and do some clever routines on
                  making sure that the map id

                  > > (e.g. 6340????.img) stay correct. Even more
                  complicated to have

                  > > consequent map id...

                  > >

                  > > For europe with a fixed max-node I get tiles
                  from 1.9MB up to 18MB.

                  > > That's factor 9 - so it's huge...

                  > > If I could narrows that down easily to
                  8-18MB - without getting tiles

                  > > crashing due to too high max-nodes values,
                  that would be sweet.

                  > >

                  > >

                  > >

                  > > As for the scripts - would they run on
                  windows too? - What programming

                  > > language are they in?

                  > >

                  > >

                  > > On 29.04.2014 21:39, osm at na1400.info
                  wrote:

                  > >> Oh, and ofcourse anyone interested can
                  get my scripts, send an email.

                  > >> They'll be on Github someday anyway.

                  > >>

                  > >>

                  > >> On 2014-04-29 20:37, Gerd Petermann
                  wrote:

                  > >>> Hi Lambertus,

                  > >>>

                  > >>> okay, if I got that right you
                  finally get *.img files with a size

                  > >>> near (but below) 8MB, so maybe
                  Henning can use that script, too.

                  > >>>

                  > >>> If you do that for e.g. Germany, how
                  small is tpically the smallest

                  > >>> *.img file ?

                  > >>> Is it probably near 4 MB?

                  > >>>

                  > >>> Gerd

                  > >>>

                  > >>>> To: mkgmap-dev at lists.mkgmap.org.uk

                  > >>>> Date: Tue, 29 Apr 2014 20:30:27
                  +0200

                  > >>>> From: osm at na1400.info

                  > >>>> Subject: Re: [mkgmap-dev] mkgmap
                  ToDo list

                  > >>>>

                  > >>>> These are the direct results
                  from Splitter. The format is o5m, both

                  > >>>> input as output. Splitter
                  version is: r321.

                  > >>>>

                  > >>>> For this test I split the
                  original source with --max-nodes=8000000.

                  > >>> Then

                  > >>>> I render the initial tiles, when
                  the result is larger than 8MB it's

                  > >>>> subsplit again with
                  --max-nodes=(8000000-(attempt*100000)). The

                  > >>> initial

                  > >>>> source files are ~70MB (o5m) and
                  after several subsplits the two

                  > >>> *.img

                  > >>>> are < 8MB. During this
                  process --max-nodes has been reduced to e.g.

                  > >>>> 7500000 and the source file is
                  split up in two o5m files of about

                  > >>> 37MB.

                  > >>>>

                  > >>>> I can upload an example source
                  file and it's two subsplit siblings

                  > >>> if

                  > >>>> you want.

                  > >>>>

                  > >>>>

                  > >>>> On 2014-04-29 19:38, GerdP
                  wrote:

                  > >>>> > Hi Lambertus,

                  > >>>> >

                  > >>>> > that's interesting. Are
                  these the img file sizes or the osm file

                  > >>> sizes?

                  > >>>> >

                  > >>>> > Gerd

                  > >>>> >

                  > >>>> >

                  > >>>> > Lambertus wrote

                  > >>>> >> Unfortunately I cannot
                  confirm that. Below is a bit of logging

                  > >>> from my

                  > >>>> >> script:

                  > >>>> >> Original: 97000020
                  (70551453), New: 0 (35684445), New: 1

                  > >>> (36852845)

                  > >>>> >> Original: 97000001
                  (74621042), New: 0 (37522992), New: 1

                  > >>> (37222739)

                  > >>>> >> Original: 97000002
                  (73391358), New: 0 (37679505), New: 1

                  > >>> (38098627)

                  > >>>> >> Original: 97000003
                  (77862567), New: 0 (39075311), New: 1

                  > >>> (39261197)

                  > >>>> >>

                  > >>>> >> The original files
                  above contain contour data, the filesize is

                  > >>> between

                  > >>>> >> brackets. As you can
                  see both resulting file are approximately

                  > >>> the

                  > >>>> >> same

                  > >>>> >> size.

                  > >>>> >>

                  > >>>> >> On 2014-04-29 15:39,
                  Gerd Petermann wrote:

                  > >>>> >>> Hi Lambertus,

                  > >>>> >>>

                  > >>>> >>> and I guess that
                  even after this optimization you will

                  > >>>> >>> see a factor 3 or
                  higher between the largest tile and the

                  > >>> smallest.

                  > >>>> >>> Can you confirm
                  that?

                  > >>>> >>>

                  > >>>> >>> Gerd

                  > >>>> >>>

                  > >>>> >>>> Date: Tue, 29
                  Apr 2014 15:32:38 +0200

                  > >>>> >>>> From:

                  > >>>> >

                  > >>>> >> osm@

                  > >>>> >

                  > >>>> >>>> To:

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> Subject: Re:
                  [mkgmap-dev] mkgmap ToDo list

                  > >>>> >>>>

                  > >>>> >>>> Num-tiles=x
                  would indeed be better for this specific need.

                  > >>>> >>>>

                  > >>>> >>>> It is my
                  experience that it regularly takes multiple calls to

                  > >>>> >>> Splitter

                  > >>>> >>>> to get 2+
                  sub-tiles when you reduce the max-nodes by 100k for

                  > >>> each

                  > >>>> >>>> sub-split
                  attempt. This is what I currently do to get an

                  > >>> optimum in

                  > >>>> >>>> tile-size vs
                  total number of tiles.

                  > >>>> >>>>

                  > >>>> >>>>

                  > >>>> >>>>

                  > >>>> >>>> On 29/04/2014
                  15:09, Gerd Petermann wrote:

                  > >>>> >>>> > Hi
                  Lambertus,

                  > >>>> >>>> >

                  > >>>> >>>> > that
                  sounds like a possible change in splitter:

                  > >>>> >>>> > Instead of
                  specifying max-nodes you may specify --num-tiles=x

                  > >>>> >>>> > and
                  splitter will try to find a split that produces
                  excactly

                  > >>> x

                  > >>>> >>> tiles

                  > >>>> >>>> > which are
                  not too narrow and have a node number which is not

                  > >>>> >>>> > too far
                  from the average (but still aligned to a multiple of

                  > >>> map

                  > >>>> >>> units

                  > >>>> >>>> > as now).

                  > >>>> >>>> > So, for
                  your script that means you don't have to find the

                  > >>>> >>> max-nodes

                  > >>>> >>>> > value.

                  > >>>> >>>> >

                  > >>>> >>>> > I'll think
                  about this again...

                  > >>>> >>>> >

                  > >>>> >>>> > Gerd

                  > >>>> >>>> >

                  > >>>> >>>> > > Date:
                  Tue, 29 Apr 2014 14:59:36 +0200

                  > >>>> >>>> > > From:

                  > >>>> >

                  > >>>> >> osm@

                  > >>>> >

                  > >>>> >>>> > > To:

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> > >
                  Subject: Re: [mkgmap-dev] mkgmap ToDo list

                  > >>>> >>>> > >

                  > >>>> >>>> > > While
                  this possibly can be solved in Splitter or Mkgmap, it

                  > >>>> >>> could also

                  > >>>> >>>> > > be
                  solved by your build-script when you add a maximum
                  tile

                  > >>> size

                  > >>>> >>> check

                  > >>>> >>>> > > and
                  re-split (with a lower number of max-nodes) until you

                  > >>> get

                  > >>>> >>> two or

                  > >>>> >>>> > > more
                  sub-tiles. Granted, this adds complexity to the script

                  > >>> but

                  > >>>> >>> it works

                  > >>>> >>>> > > well
                  for me.

                  > >>>> >>>> > >

                  > >>>> >>>> > > On
                  25/04/2014 21:54, Henning Scholland wrote:

                  > >>>> >>>> > > >
                  Hi Gerd,

                  > >>>> >>>> > > >

                  > >>>> >>>> > > >
                  I would like to have img-tiles which have globally
                  nearly

                  > >>> the

                  > >>>> >>> same

                  > >>>> >>>> > > >
                  filesize, so that they use the space of devices like

                  > >>> eTrex 10.

                  > >>>> >>>> > > >

                  > >>>> >>>> > > >
                  With my actual map I use globally the same value for

                  > >>>> >>> max-nodes. But the

                  > >>>> >>>> > > >
                  size of the img-tiles differ more then factor 2. Eg. a

                  > >>> tile in

                  > >>>> >>> Germany

                  > >>>> >>>> > > >
                  is between 2 and 5 mb where a tile in China is about
                  10

                  > >>> mb. If

                  > >>>> >>> I remove

                  > >>>> >>>> > > >
                  details, this difference will increase, because in

                  > >>> Germany

                  > >>>> >>> more objects

                  > >>>> >>>> > > >
                  will be removed from the img-tile then in China.

                  > >>>> >>>> > > >

                  > >>>> >>>> > > >
                  Henning

                  > >>>> >>>> > > >

                  > >>>> >>>> > > >
                  _______________________________________________

                  > >>>> >>>> > > >
                  mkgmap-dev mailing list

                  > >>>> >>>> > > >

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> > > >
                  http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >>>> > >

                  > >>>> >>>> > >
                  _______________________________________________

                  > >>>> >>>> > >
                  mkgmap-dev mailing list

                  > >>>> >>>> > >

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >>>> >

                  > >>>> >>>> >

                  > >>>> >>>> >
                  _______________________________________________

                  > >>>> >>>> > mkgmap-dev
                  mailing list

                  > >>>> >>>> >

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >>>> >

                  > >>>> >>>>

                  > >>>> >>>>
                  _______________________________________________

                  > >>>> >>>> mkgmap-dev
                  mailing list

                  > >>>> >>>>

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >>>

                  > >>>> >>>
                  _______________________________________________

                  > >>>> >>> mkgmap-dev mailing
                  list

                  > >>>> >>>

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >>
                  _______________________________________________

                  > >>>> >> mkgmap-dev mailing list

                  > >>>> >

                  > >>>> >> mkgmap-dev at .org

                  > >>>> >

                  > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>> >

                  > >>>> >

                  > >>>> >

                  > >>>> >

                  > >>>> >

                  > >>>> > --

                  > >>>> > View this message in
                  context:

                  > >>>> >

                  > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html

                  > >>>> > Sent from the Mkgmap
                  Development mailing list archive at

                  > >>> Nabble.com.

                  > >>>> >
                  _______________________________________________

                  > >>>> > mkgmap-dev mailing list

                  > >>>> > mkgmap-dev at lists.mkgmap.org.uk

                  > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>>
                  _______________________________________________

                  > >>>> mkgmap-dev mailing list

                  > >>>> mkgmap-dev at lists.mkgmap.org.uk

                  > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>>

                  > >>>
                  _______________________________________________

                  > >>> mkgmap-dev mailing list

                  > >>> mkgmap-dev at lists.mkgmap.org.uk

                  > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >>
                  _______________________________________________

                  > >> mkgmap-dev mailing list

                  > >> mkgmap-dev at lists.mkgmap.org.uk

                  > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                  > >

                  > 

                  > _______________________________________________

                  > mkgmap-dev mailing list

                  > mkgmap-dev at lists.mkgmap.org.uk

                  > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

                
              
            
            

            
            

            _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
          
          

          -- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org
          

          _______________________________________________
          mkgmap-dev mailing list
          mkgmap-dev at lists.mkgmap.org.uk
          http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
      
      

      
      

      _______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
    
    

    -- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org
  


_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140505/3d308a4c/attachment-0001.html>


More information about the mkgmap-dev mailing list