[mkgmap-dev] mkgmap ToDo list

Wed Apr 30 12:36:19 BST 2014

Multithreading the tile rendering for a single map is indeed a bit 
difficult and I gave it up because you need to keep track which image 
id's are already in use. Since I provide multiple maps the work-around 
is running a few scripts parallel, which is also a crude form of 
multithreading.

The script language is PHP and it doesn't run on Windows without some 
changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it.

To get a better optimum in file size, using the process I described 
earlier, you could start off with a huge --max-nodes setting and then 
'search' for the highest --max-nodes that works for each specific area.

On 30/04/2014 11:49, Felix Hartmann wrote:
> I would love if there was a possibility that you pass the used max-nodes
> value to mkgmap.
> When mkgmap is compiling the maps, then after the .img is created it
> should check
> a) did it crash due to too many max-nodes
> b) for me not important - but for others with very old GPS, etrex 10,
> ---> is tile bigger than X (usually 8) MB.
>
> if a) or b) true, then pass the file back to splitter and split with 60%
> of maxnodes - and compile the resulting .img files again. Should it fail
> again, use 40%, again 25%... Sometimes there are awful tiles, that need
> supersmall max-nodes till they compile, however lately (last 1-2 years)
> I never encountered them anymore. I think that happened rather due to a
> but in splitter/mgkmap that is fixed now.
>
> okay, you could also do this with a script, but it gets rather
> complicated to multithread it (you need to wait till mgkmap finished
> compiling all .img files - and run mkgmap first without address index to
> save time) and do some clever routines on making sure that the map id
> (e.g. 6340????.img)  stay correct. Even more complicated to have
> consequent map id...
>
> For europe with a fixed max-node I get tiles from 1.9MB up to 18MB.
> That's factor 9 - so it's huge...
> If I could narrows that down easily to 8-18MB - without getting tiles
> crashing due to too high max-nodes values, that would be sweet.
>
>
>
> As for the scripts - would they run on windows too? - What programming
> language are they in?
>
>
> On 29.04.2014 21:39, osm at na1400.info wrote:
>> Oh, and ofcourse anyone interested can get my scripts, send an email.
>> They'll be on Github someday anyway.
>>
>>
>> On 2014-04-29 20:37, Gerd Petermann wrote:
>>> Hi Lambertus,
>>>
>>> okay, if I got that right you finally get *.img files with a size
>>> near (but below) 8MB, so maybe Henning can use that script, too.
>>>
>>> If you do that for e.g. Germany, how small is tpically the smallest
>>> *.img file ?
>>> Is it probably near 4 MB?
>>>
>>> Gerd
>>>
>>>> To: mkgmap-dev at lists.mkgmap.org.uk
>>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
>>>> From: osm at na1400.info
>>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>>
>>>> These are the direct results from Splitter. The format is o5m, both
>>>> input as output. Splitter version is: r321.
>>>>
>>>> For this test I split the original source with --max-nodes=8000000.
>>> Then
>>>> I render the initial tiles, when the result is larger than 8MB it's
>>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The
>>> initial
>>>> source files are ~70MB (o5m) and after several subsplits the two
>>> *.img
>>>> are < 8MB. During this process --max-nodes has been reduced to e.g.
>>>> 7500000 and the source file is split up in two o5m files of about
>>> 37MB.
>>>>
>>>> I can upload an example source file and it's two subsplit siblings
>>> if
>>>> you want.
>>>>
>>>>
>>>> On 2014-04-29 19:38, GerdP wrote:
>>>> > Hi Lambertus,
>>>> >
>>>> > that's interesting. Are these the img file sizes or the osm file
>>> sizes?
>>>> >
>>>> > Gerd
>>>> >
>>>> >
>>>> > Lambertus wrote
>>>> >> Unfortunately I cannot confirm that. Below is a bit of logging
>>> from my
>>>> >> script:
>>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
>>> (36852845)
>>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
>>> (37222739)
>>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
>>> (38098627)
>>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
>>> (39261197)
>>>> >>
>>>> >> The original files above contain contour data, the filesize is
>>> between
>>>> >> brackets. As you can see both resulting file are approximately
>>> the
>>>> >> same
>>>> >> size.
>>>> >>
>>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
>>>> >>> Hi Lambertus,
>>>> >>>
>>>> >>> and I guess that even after this optimization you will
>>>> >>> see a factor 3 or higher between the largest tile and the
>>> smallest.
>>>> >>> Can you confirm that?
>>>> >>>
>>>> >>> Gerd
>>>> >>>
>>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
>>>> >>>> From:
>>>> >
>>>> >> osm@
>>>> >
>>>> >>>> To:
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> >>>>
>>>> >>>> Num-tiles=x would indeed be better for this specific need.
>>>> >>>>
>>>> >>>> It is my experience that it regularly takes multiple calls to
>>>> >>> Splitter
>>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for
>>> each
>>>> >>>> sub-split attempt. This is what I currently do to get an
>>> optimum in
>>>> >>>> tile-size vs total number of tiles.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
>>>> >>>> > Hi Lambertus,
>>>> >>>> >
>>>> >>>> > that sounds like a possible change in splitter:
>>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x
>>>> >>>> > and splitter will try to find a split that produces excactly
>>> x
>>>> >>> tiles
>>>> >>>> > which are not too narrow and have a node number which is not
>>>> >>>> > too far from the average (but still aligned to a multiple of
>>> map
>>>> >>> units
>>>> >>>> > as now).
>>>> >>>> > So, for your script that means you don't have to find the
>>>> >>> max-nodes
>>>> >>>> > value.
>>>> >>>> >
>>>> >>>> > I'll think about this again...
>>>> >>>> >
>>>> >>>> > Gerd
>>>> >>>> >
>>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
>>>> >>>> > > From:
>>>> >
>>>> >> osm@
>>>> >
>>>> >>>> > > To:
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> >>>> > >
>>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it
>>>> >>> could also
>>>> >>>> > > be solved by your build-script when you add a maximum tile
>>> size
>>>> >>> check
>>>> >>>> > > and re-split (with a lower number of max-nodes) until you
>>> get
>>>> >>> two or
>>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script
>>> but
>>>> >>> it works
>>>> >>>> > > well for me.
>>>> >>>> > >
>>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
>>>> >>>> > > > Hi Gerd,
>>>> >>>> > > >
>>>> >>>> > > > I would like to have img-tiles which have globally nearly
>>> the
>>>> >>> same
>>>> >>>> > > > filesize, so that they use the space of devices like
>>> eTrex 10.
>>>> >>>> > > >
>>>> >>>> > > > With my actual map I use globally the same value for
>>>> >>> max-nodes. But the
>>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a
>>> tile in
>>>> >>> Germany
>>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10
>>> mb. If
>>>> >>> I remove
>>>> >>>> > > > details, this difference will increase, because in
>>> Germany
>>>> >>> more objects
>>>> >>>> > > > will be removed from the img-tile then in China.
>>>> >>>> > > >
>>>> >>>> > > > Henning
>>>> >>>> > > >
>>>> >>>> > > > _______________________________________________
>>>> >>>> > > > mkgmap-dev mailing list
>>>> >>>> > > >
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >>>> > >
>>>> >>>> > > _______________________________________________
>>>> >>>> > > mkgmap-dev mailing list
>>>> >>>> > >
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > _______________________________________________
>>>> >>>> > mkgmap-dev mailing list
>>>> >>>> >
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >>>> >
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>> mkgmap-dev mailing list
>>>> >>>>
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> mkgmap-dev mailing list
>>>> >>>
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >> _______________________________________________
>>>> >> mkgmap-dev mailing list
>>>> >
>>>> >> mkgmap-dev at .org
>>>> >
>>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > View this message in context:
>>>> >
>>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
>>>> > Sent from the Mkgmap Development mailing list archive at
>>> Nabble.com.
>>>> > _______________________________________________
>>>> > mkgmap-dev mailing list
>>>> > mkgmap-dev at lists.mkgmap.org.uk
>>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>> _______________________________________________
>>>> mkgmap-dev mailing list
>>>> mkgmap-dev at lists.mkgmap.org.uk
>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>