[mkgmap-dev] mkgmap ToDo list
From Felix Hartmann extremecarver at gmail.com on Mon May 5 12:36:21 BST 2014
Well - I suppose that is only for continuing to split. Not like saying I want to split europe and --num-tiles=600 or? Anyhow that's very good in case mkgmap would pass back osm.pbf files to splitter that failed compiling. Because then it could pass forward - "split into 2 tiles" :-) Or better -there is no real change in the splitting algo behind I assume on counting max-nodes or whatever else.. On 05.05.2014 09:28, Gerd Petermann wrote: > Hi Lambertus, > > attached is a patch that implements the new option: > --num-tiles A target value that is used when no split-file is > given. > Splitting is done so that the given number of tiles is > produced. The max-nodes value is ignored if this > option is > given. > A compiled binary is here: > > |http://files.mkgmap.org.uk/download/206/splitter.jar > > Please let me know if it works as expected. > > Gerd > | > ------------------------------------------------------------------------ > Date: Sun, 4 May 2014 22:20:20 +0200 > From: osm at na1400.info > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Great, many thanks. I'm currently trying to create as large as > (automatically) possible contour tiles and a lot of time is put in > subsplitting a tile again and again until two subtiles appear. This > new option would save much time. > > On 05/04/2014 10:13 PM, Gerd Petermann wrote: > > Hi Lambertus, > > yes, I want to code that tomorrow. I prefer to make it "give me n > tiles", but if that turns out to be > difficult I try the simple version. I can think of scenarios where > it is not possible to create exactly > n tiles. > > Gerd > > ------------------------------------------------------------------------ > Date: Sun, 4 May 2014 22:08:32 +0200 > From: osm at na1400.info <mailto:osm at na1400.info> > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Is the 'just give me two tiles' option for splitter on the to-do > list? I'd really appreciate such a function. > > On 05/04/2014 09:36 PM, Gerd Petermann wrote: > > Hi Felix, > > > would using the precomp-sea parameter for splitter improve > results? It's not that tiles with loads of sea actually failed. > > good question. I've coded that parameter in splitter before we > did some important changes in mkgmap. > I think it is needed when you use a polygon file in > combination with an imput file that doesn't cover the polygon. > If the polygon covers a costline with a lot of islands, but > the input file doesn't contain the data, > splitter might create large tiles covering these "empty" > areas. Later, in mkgmap, the empty > areas are filled with the precompiled-sea data, and that can > cause too large files. > > BTW: The quick hack turned out to be useless. It worked better > in some areas and worse in others. > I'll look again at this later next week. > > Gerd > > > > ------------------------------------------------------------------------ > Date: Sun, 4 May 2014 21:21:59 +0200 > From: extremecarver at gmail.com <mailto:extremecarver at gmail.com> > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > No - I use splitter like this - with maxnodes depending on > area - I usually just reduced it after failures to compile a > tile.: > java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar > --max-nodes=%maxnodes% --max-threads=8 --output=pbf > --keep-complete --max-areas=1524 --geonames-file=cities5000 > --description=%country% --mapid=%FID%0000 > c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf > would using the precomp-sea parameter for splitter improve > results? It's not that tiles with loads of sea actually failed. > > On 03.05.2014 09:00, Gerd Petermann wrote: > > Hi all, > > I've coded a quick hack which seems to improve the ratios. > On the other hand, I don't see these large differences between > smallest and largest img file. > What part of the world should I try? > Do you use the precomp-sea parameter in splitter? > > Gerd > > ------------------------------------------------------------------------ > Date: Wed, 30 Apr 2014 14:59:32 +0200 > From: extremecarver at gmail.com <mailto:extremecarver at gmail.com> > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > if that doesn't seriously (more than 30-40%) slow down the > splitter, I assume it would be much better... > On 30.04.2014 14:06, Gerd Petermann wrote: > > Hi, > > if I got that right the number of nodes is not > highly correlated to the img size, so the max-nodes > value is not a good estimate. > > I assume the reason is that nodes which belong to > roads produce a lot more bytes > in the img file compared to nodes which are parts > of shapes or other non-routable ways, not talking > about nodes which are simply ignored by the style. > > So, a possible solution in splitter could be to parse > the ways before reading the nodes and save all nodeids > which belong to ways with highway=*. > If these nodes are refered by more than one way with > highway=* > we assume that they will be routing nodes. > These special nodes could be counted e.g. 10 times to > give a better estimate. > > Gerd > > > Date: Wed, 30 Apr 2014 13:36:19 +0200 > > From: osm at na1400.info <mailto:osm at na1400.info> > > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > Multithreading the tile rendering for a single map > is indeed a bit > > difficult and I gave it up because you need to keep > track which image > > id's are already in use. Since I provide multiple > maps the work-around > > is running a few scripts parallel, which is also a > crude form of > > multithreading. > > > > The script language is PHP and it doesn't run on > Windows without some > > changes ('/' vs '\' in paths, 'rm -rf', that sort of > stuff). Never tried it. > > > > To get a better optimum in file size, using the > process I described > > earlier, you could start off with a huge --max-nodes > setting and then > > 'search' for the highest --max-nodes that works for > each specific area. > > > > On 30/04/2014 11:49, Felix Hartmann wrote: > > > I would love if there was a possibility that you > pass the used max-nodes > > > value to mkgmap. > > > When mkgmap is compiling the maps, then after the > .img is created it > > > should check > > > a) did it crash due to too many max-nodes > > > b) for me not important - but for others with very > old GPS, etrex 10, > > > ---> is tile bigger than X (usually 8) MB. > > > > > > if a) or b) true, then pass the file back to > splitter and split with 60% > > > of maxnodes - and compile the resulting .img files > again. Should it fail > > > again, use 40%, again 25%... Sometimes there are > awful tiles, that need > > > supersmall max-nodes till they compile, however > lately (last 1-2 years) > > > I never encountered them anymore. I think that > happened rather due to a > > > but in splitter/mgkmap that is fixed now. > > > > > > okay, you could also do this with a script, but it > gets rather > > > complicated to multithread it (you need to wait > till mgkmap finished > > > compiling all .img files - and run mkgmap first > without address index to > > > save time) and do some clever routines on making > sure that the map id > > > (e.g. 6340????.img) stay correct. Even more > complicated to have > > > consequent map id... > > > > > > For europe with a fixed max-node I get tiles from > 1.9MB up to 18MB. > > > That's factor 9 - so it's huge... > > > If I could narrows that down easily to 8-18MB - > without getting tiles > > > crashing due to too high max-nodes values, that > would be sweet. > > > > > > > > > > > > As for the scripts - would they run on windows > too? - What programming > > > language are they in? > > > > > > > > > On 29.04.2014 21:39, osm at na1400.info > <mailto:osm at na1400.info> wrote: > > >> Oh, and ofcourse anyone interested can get my > scripts, send an email. > > >> They'll be on Github someday anyway. > > >> > > >> > > >> On 2014-04-29 20:37, Gerd Petermann wrote: > > >>> Hi Lambertus, > > >>> > > >>> okay, if I got that right you finally get *.img > files with a size > > >>> near (but below) 8MB, so maybe Henning can use > that script, too. > > >>> > > >>> If you do that for e.g. Germany, how small is > tpically the smallest > > >>> *.img file ? > > >>> Is it probably near 4 MB? > > >>> > > >>> Gerd > > >>> > > >>>> To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > > >>>> From: osm at na1400.info <mailto:osm at na1400.info> > > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> > > >>>> These are the direct results from Splitter. The > format is o5m, both > > >>>> input as output. Splitter version is: r321. > > >>>> > > >>>> For this test I split the original source with > --max-nodes=8000000. > > >>> Then > > >>>> I render the initial tiles, when the result is > larger than 8MB it's > > >>>> subsplit again with > --max-nodes=(8000000-(attempt*100000)). The > > >>> initial > > >>>> source files are ~70MB (o5m) and after several > subsplits the two > > >>> *.img > > >>>> are < 8MB. During this process --max-nodes has > been reduced to e.g. > > >>>> 7500000 and the source file is split up in two > o5m files of about > > >>> 37MB. > > >>>> > > >>>> I can upload an example source file and it's > two subsplit siblings > > >>> if > > >>>> you want. > > >>>> > > >>>> > > >>>> On 2014-04-29 19:38, GerdP wrote: > > >>>> > Hi Lambertus, > > >>>> > > > >>>> > that's interesting. Are these the img file > sizes or the osm file > > >>> sizes? > > >>>> > > > >>>> > Gerd > > >>>> > > > >>>> > > > >>>> > Lambertus wrote > > >>>> >> Unfortunately I cannot confirm that. Below > is a bit of logging > > >>> from my > > >>>> >> script: > > >>>> >> Original: 97000020 (70551453), New: 0 > (35684445), New: 1 > > >>> (36852845) > > >>>> >> Original: 97000001 (74621042), New: 0 > (37522992), New: 1 > > >>> (37222739) > > >>>> >> Original: 97000002 (73391358), New: 0 > (37679505), New: 1 > > >>> (38098627) > > >>>> >> Original: 97000003 (77862567), New: 0 > (39075311), New: 1 > > >>> (39261197) > > >>>> >> > > >>>> >> The original files above contain contour > data, the filesize is > > >>> between > > >>>> >> brackets. As you can see both resulting file > are approximately > > >>> the > > >>>> >> same > > >>>> >> size. > > >>>> >> > > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > > >>>> >>> Hi Lambertus, > > >>>> >>> > > >>>> >>> and I guess that even after this > optimization you will > > >>>> >>> see a factor 3 or higher between the > largest tile and the > > >>> smallest. > > >>>> >>> Can you confirm that? > > >>>> >>> > > >>>> >>> Gerd > > >>>> >>> > > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > > >>>> >>>> From: > > >>>> > > > >>>> >> osm@ > > >>>> > > > >>>> >>>> To: > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> >>>> > > >>>> >>>> Num-tiles=x would indeed be better for > this specific need. > > >>>> >>>> > > >>>> >>>> It is my experience that it regularly > takes multiple calls to > > >>>> >>> Splitter > > >>>> >>>> to get 2+ sub-tiles when you reduce the > max-nodes by 100k for > > >>> each > > >>>> >>>> sub-split attempt. This is what I > currently do to get an > > >>> optimum in > > >>>> >>>> tile-size vs total number of tiles. > > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > > >>>> >>>> > Hi Lambertus, > > >>>> >>>> > > > >>>> >>>> > that sounds like a possible change in > splitter: > > >>>> >>>> > Instead of specifying max-nodes you may > specify --num-tiles=x > > >>>> >>>> > and splitter will try to find a split > that produces excactly > > >>> x > > >>>> >>> tiles > > >>>> >>>> > which are not too narrow and have a node > number which is not > > >>>> >>>> > too far from the average (but still > aligned to a multiple of > > >>> map > > >>>> >>> units > > >>>> >>>> > as now). > > >>>> >>>> > So, for your script that means you don't > have to find the > > >>>> >>> max-nodes > > >>>> >>>> > value. > > >>>> >>>> > > > >>>> >>>> > I'll think about this again... > > >>>> >>>> > > > >>>> >>>> > Gerd > > >>>> >>>> > > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > >>>> >>>> > > From: > > >>>> > > > >>>> >> osm@ > > >>>> > > > >>>> >>>> > > To: > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> >>>> > > > > >>>> >>>> > > While this possibly can be solved in > Splitter or Mkgmap, it > > >>>> >>> could also > > >>>> >>>> > > be solved by your build-script when > you add a maximum tile > > >>> size > > >>>> >>> check > > >>>> >>>> > > and re-split (with a lower number of > max-nodes) until you > > >>> get > > >>>> >>> two or > > >>>> >>>> > > more sub-tiles. Granted, this adds > complexity to the script > > >>> but > > >>>> >>> it works > > >>>> >>>> > > well for me. > > >>>> >>>> > > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland > wrote: > > >>>> >>>> > > > Hi Gerd, > > >>>> >>>> > > > > > >>>> >>>> > > > I would like to have img-tiles which > have globally nearly > > >>> the > > >>>> >>> same > > >>>> >>>> > > > filesize, so that they use the space > of devices like > > >>> eTrex 10. > > >>>> >>>> > > > > > >>>> >>>> > > > With my actual map I use globally > the same value for > > >>>> >>> max-nodes. But the > > >>>> >>>> > > > size of the img-tiles differ more > then factor 2. Eg. a > > >>> tile in > > >>>> >>> Germany > > >>>> >>>> > > > is between 2 and 5 mb where a tile > in China is about 10 > > >>> mb. If > > >>>> >>> I remove > > >>>> >>>> > > > details, this difference will > increase, because in > > >>> Germany > > >>>> >>> more objects > > >>>> >>>> > > > will be removed from the img-tile > then in China. > > >>>> >>>> > > > > > >>>> >>>> > > > Henning > > >>>> >>>> > > > > > >>>> >>>> > > > > _______________________________________________ > > >>>> >>>> > > > mkgmap-dev mailing list > > >>>> >>>> > > > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > > >>>> >>>> > > mkgmap-dev mailing list > > >>>> >>>> > > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > > >>>> >>>> > mkgmap-dev mailing list > > >>>> >>>> > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > > >>>> >>>> mkgmap-dev mailing list > > >>>> >>>> > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>> > > >>>> >>> _______________________________________________ > > >>>> >>> mkgmap-dev mailing list > > >>>> >>> > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >> _______________________________________________ > > >>>> >> mkgmap-dev mailing list > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > -- > > >>>> > View this message in context: > > >>>> > > > >>> > http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > > >>>> > Sent from the Mkgmap Development mailing list > archive at > > >>> Nabble.com. > > >>>> > _______________________________________________ > > >>>> > mkgmap-dev mailing list > > >>>> > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> _______________________________________________ > > >>>> mkgmap-dev mailing list > > >>>> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>> > > >>> _______________________________________________ > > >>> mkgmap-dev mailing list > > >>> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >> _______________________________________________ > > >> mkgmap-dev mailing list > > >> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > _______________________________________________ mkgmap-dev > mailing list mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > _______________________________________________ mkgmap-dev > mailing list mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ mkgmap-dev mailing > list mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ mkgmap-dev mailing > list mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140505/e5a738cf/attachment-0001.html>
- Previous message: [mkgmap-dev] mkgmap ToDo list
- Next message: [mkgmap-dev] mkgmap ToDo list
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list