[mkgmap-dev] mkgmap ToDo list

Wed Apr 30 13:06:34 BST 2014

Hi,

if I got that right the number of nodes is not 
highly correlated to the img size, so the max-nodes
value  is not a good estimate.

I assume the reason is that nodes which belong to
roads produce a lot more bytes
in the img file compared to nodes which are parts
of shapes or other non-routable ways, not talking
about nodes which are simply ignored by the style.

So, a possible solution in splitter could be to parse 
the ways before reading the nodes and save all nodeids
which belong to ways with highway=*.
If these nodes are refered by more than one way with highway=*
we assume that they will be routing nodes.
These special nodes could be counted e.g. 10 times to 
give a better estimate.

Gerd

> Date: Wed, 30 Apr 2014 13:36:19 +0200
> From: osm at na1400.info
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> 
> Multithreading the tile rendering for a single map is indeed a bit 
> difficult and I gave it up because you need to keep track which image 
> id's are already in use. Since I provide multiple maps the work-around 
> is running a few scripts parallel, which is also a crude form of 
> multithreading.
> 
> The script language is PHP and it doesn't run on Windows without some 
> changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it.
> 
> To get a better optimum in file size, using the process I described 
> earlier, you could start off with a huge --max-nodes setting and then 
> 'search' for the highest --max-nodes that works for each specific area.
> 
> On 30/04/2014 11:49, Felix Hartmann wrote:
> > I would love if there was a possibility that you pass the used max-nodes
> > value to mkgmap.
> > When mkgmap is compiling the maps, then after the .img is created it
> > should check
> > a) did it crash due to too many max-nodes
> > b) for me not important - but for others with very old GPS, etrex 10,
> > ---> is tile bigger than X (usually 8) MB.
> >
> > if a) or b) true, then pass the file back to splitter and split with 60%
> > of maxnodes - and compile the resulting .img files again. Should it fail
> > again, use 40%, again 25%... Sometimes there are awful tiles, that need
> > supersmall max-nodes till they compile, however lately (last 1-2 years)
> > I never encountered them anymore. I think that happened rather due to a
> > but in splitter/mgkmap that is fixed now.
> >
> > okay, you could also do this with a script, but it gets rather
> > complicated to multithread it (you need to wait till mgkmap finished
> > compiling all .img files - and run mkgmap first without address index to
> > save time) and do some clever routines on making sure that the map id
> > (e.g. 6340????.img)  stay correct. Even more complicated to have
> > consequent map id...
> >
> > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB.
> > That's factor 9 - so it's huge...
> > If I could narrows that down easily to 8-18MB - without getting tiles
> > crashing due to too high max-nodes values, that would be sweet.
> >
> >
> >
> > As for the scripts - would they run on windows too? - What programming
> > language are they in?
> >
> >
> > On 29.04.2014 21:39, osm at na1400.info wrote:
> >> Oh, and ofcourse anyone interested can get my scripts, send an email.
> >> They'll be on Github someday anyway.
> >>
> >>
> >> On 2014-04-29 20:37, Gerd Petermann wrote:
> >>> Hi Lambertus,
> >>>
> >>> okay, if I got that right you finally get *.img files with a size
> >>> near (but below) 8MB, so maybe Henning can use that script, too.
> >>>
> >>> If you do that for e.g. Germany, how small is tpically the smallest
> >>> *.img file ?
> >>> Is it probably near 4 MB?
> >>>
> >>> Gerd
> >>>
> >>>> To: mkgmap-dev at lists.mkgmap.org.uk
> >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
> >>>> From: osm at na1400.info
> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> >>>>
> >>>> These are the direct results from Splitter. The format is o5m, both
> >>>> input as output. Splitter version is: r321.
> >>>>
> >>>> For this test I split the original source with --max-nodes=8000000.
> >>> Then
> >>>> I render the initial tiles, when the result is larger than 8MB it's
> >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The
> >>> initial
> >>>> source files are ~70MB (o5m) and after several subsplits the two
> >>> *.img
> >>>> are < 8MB. During this process --max-nodes has been reduced to e.g.
> >>>> 7500000 and the source file is split up in two o5m files of about
> >>> 37MB.
> >>>>
> >>>> I can upload an example source file and it's two subsplit siblings
> >>> if
> >>>> you want.
> >>>>
> >>>>
> >>>> On 2014-04-29 19:38, GerdP wrote:
> >>>> > Hi Lambertus,
> >>>> >
> >>>> > that's interesting. Are these the img file sizes or the osm file
> >>> sizes?
> >>>> >
> >>>> > Gerd
> >>>> >
> >>>> >
> >>>> > Lambertus wrote
> >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging
> >>> from my
> >>>> >> script:
> >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
> >>> (36852845)
> >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
> >>> (37222739)
> >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
> >>> (38098627)
> >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
> >>> (39261197)
> >>>> >>
> >>>> >> The original files above contain contour data, the filesize is
> >>> between
> >>>> >> brackets. As you can see both resulting file are approximately
> >>> the
> >>>> >> same
> >>>> >> size.
> >>>> >>
> >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
> >>>> >>> Hi Lambertus,
> >>>> >>>
> >>>> >>> and I guess that even after this optimization you will
> >>>> >>> see a factor 3 or higher between the largest tile and the
> >>> smallest.
> >>>> >>> Can you confirm that?
> >>>> >>>
> >>>> >>> Gerd
> >>>> >>>
> >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
> >>>> >>>> From:
> >>>> >
> >>>> >> osm@
> >>>> >
> >>>> >>>> To:
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> >>>> >>>>
> >>>> >>>> Num-tiles=x would indeed be better for this specific need.
> >>>> >>>>
> >>>> >>>> It is my experience that it regularly takes multiple calls to
> >>>> >>> Splitter
> >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for
> >>> each
> >>>> >>>> sub-split attempt. This is what I currently do to get an
> >>> optimum in
> >>>> >>>> tile-size vs total number of tiles.
> >>>> >>>>
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
> >>>> >>>> > Hi Lambertus,
> >>>> >>>> >
> >>>> >>>> > that sounds like a possible change in splitter:
> >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x
> >>>> >>>> > and splitter will try to find a split that produces excactly
> >>> x
> >>>> >>> tiles
> >>>> >>>> > which are not too narrow and have a node number which is not
> >>>> >>>> > too far from the average (but still aligned to a multiple of
> >>> map
> >>>> >>> units
> >>>> >>>> > as now).
> >>>> >>>> > So, for your script that means you don't have to find the
> >>>> >>> max-nodes
> >>>> >>>> > value.
> >>>> >>>> >
> >>>> >>>> > I'll think about this again...
> >>>> >>>> >
> >>>> >>>> > Gerd
> >>>> >>>> >
> >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
> >>>> >>>> > > From:
> >>>> >
> >>>> >> osm@
> >>>> >
> >>>> >>>> > > To:
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
> >>>> >>>> > >
> >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it
> >>>> >>> could also
> >>>> >>>> > > be solved by your build-script when you add a maximum tile
> >>> size
> >>>> >>> check
> >>>> >>>> > > and re-split (with a lower number of max-nodes) until you
> >>> get
> >>>> >>> two or
> >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script
> >>> but
> >>>> >>> it works
> >>>> >>>> > > well for me.
> >>>> >>>> > >
> >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
> >>>> >>>> > > > Hi Gerd,
> >>>> >>>> > > >
> >>>> >>>> > > > I would like to have img-tiles which have globally nearly
> >>> the
> >>>> >>> same
> >>>> >>>> > > > filesize, so that they use the space of devices like
> >>> eTrex 10.
> >>>> >>>> > > >
> >>>> >>>> > > > With my actual map I use globally the same value for
> >>>> >>> max-nodes. But the
> >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a
> >>> tile in
> >>>> >>> Germany
> >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10
> >>> mb. If
> >>>> >>> I remove
> >>>> >>>> > > > details, this difference will increase, because in
> >>> Germany
> >>>> >>> more objects
> >>>> >>>> > > > will be removed from the img-tile then in China.
> >>>> >>>> > > >
> >>>> >>>> > > > Henning
> >>>> >>>> > > >
> >>>> >>>> > > > _______________________________________________
> >>>> >>>> > > > mkgmap-dev mailing list
> >>>> >>>> > > >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> > >
> >>>> >>>> > > _______________________________________________
> >>>> >>>> > > mkgmap-dev mailing list
> >>>> >>>> > >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> > _______________________________________________
> >>>> >>>> > mkgmap-dev mailing list
> >>>> >>>> >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> >
> >>>> >>>>
> >>>> >>>> _______________________________________________
> >>>> >>>> mkgmap-dev mailing list
> >>>> >>>>
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>
> >>>> >>> _______________________________________________
> >>>> >>> mkgmap-dev mailing list
> >>>> >>>
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >> _______________________________________________
> >>>> >> mkgmap-dev mailing list
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > View this message in context:
> >>>> >
> >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
> >>>> > Sent from the Mkgmap Development mailing list archive at
> >>> Nabble.com.
> >>>> > _______________________________________________
> >>>> > mkgmap-dev mailing list
> >>>> > mkgmap-dev at lists.mkgmap.org.uk
> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> _______________________________________________
> >>>> mkgmap-dev mailing list
> >>>> mkgmap-dev at lists.mkgmap.org.uk
> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>
> >>> _______________________________________________
> >>> mkgmap-dev mailing list
> >>> mkgmap-dev at lists.mkgmap.org.uk
> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >> _______________________________________________
> >> mkgmap-dev mailing list
> >> mkgmap-dev at lists.mkgmap.org.uk
> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140430/4db801c9/attachment-0001.html>