[mkgmap-dev] mkgmap ToDo list
From GerdP gpetermann_muenchen at hotmail.com on Mon May 5 12:49:48 BST 2014
Hi Felix, I did not try with Europe, only with Finland, but I see no reason why it should not work. I'll see what I can do regarding the automatic call of splitter from mkgmap. Gerd Felix Hartmann-2 wrote > Well - I suppose that is only for continuing to split. Not like saying I > want to split europe and --num-tiles=600 or? > Anyhow that's very good in case mkgmap would pass back osm.pbf files to > splitter that failed compiling. Because then it could pass forward - > "split into 2 tiles" :-) > > Or better -there is no real change in the splitting algo behind I assume > on counting max-nodes or whatever else.. > On 05.05.2014 09:28, Gerd Petermann wrote: >> Hi Lambertus, >> >> attached is a patch that implements the new option: >> --num-tiles A target value that is used when no split-file is >> given. >> Splitting is done so that the given number of tiles is >> produced. The max-nodes value is ignored if this >> option is >> given. >> A compiled binary is here: >> >> |http://files.mkgmap.org.uk/download/206/splitter.jar >> >> Please let me know if it works as expected. >> >> Gerd >> | >> ------------------------------------------------------------------------ >> Date: Sun, 4 May 2014 22:20:20 +0200 >> From: > osm@ >> To: > mkgmap-dev at .org >> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> Great, many thanks. I'm currently trying to create as large as >> (automatically) possible contour tiles and a lot of time is put in >> subsplitting a tile again and again until two subtiles appear. This >> new option would save much time. >> >> On 05/04/2014 10:13 PM, Gerd Petermann wrote: >> >> Hi Lambertus, >> >> yes, I want to code that tomorrow. I prefer to make it "give me n >> tiles", but if that turns out to be >> difficult I try the simple version. I can think of scenarios where >> it is not possible to create exactly >> n tiles. >> >> Gerd >> >> >> ------------------------------------------------------------------------ >> Date: Sun, 4 May 2014 22:08:32 +0200 >> From: > osm@ > <mailto: > osm@ > > >> To: > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> Is the 'just give me two tiles' option for splitter on the to-do >> list? I'd really appreciate such a function. >> >> On 05/04/2014 09:36 PM, Gerd Petermann wrote: >> >> Hi Felix, >> >> > would using the precomp-sea parameter for splitter improve >> results? It's not that tiles with loads of sea actually failed. >> >> good question. I've coded that parameter in splitter before we >> did some important changes in mkgmap. >> I think it is needed when you use a polygon file in >> combination with an imput file that doesn't cover the polygon. >> If the polygon covers a costline with a lot of islands, but >> the input file doesn't contain the data, >> splitter might create large tiles covering these "empty" >> areas. Later, in mkgmap, the empty >> areas are filled with the precompiled-sea data, and that can >> cause too large files. >> >> BTW: The quick hack turned out to be useless. It worked better >> in some areas and worse in others. >> I'll look again at this later next week. >> >> Gerd >> >> >> >> >> ------------------------------------------------------------------------ >> Date: Sun, 4 May 2014 21:21:59 +0200 >> From: > extremecarver@ > <mailto: > extremecarver@ > > >> To: > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> No - I use splitter like this - with maxnodes depending on >> area - I usually just reduced it after failures to compile a >> tile.: >> java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar >> --max-nodes=%maxnodes% --max-threads=8 --output=pbf >> --keep-complete --max-areas=1524 --geonames-file=cities5000 >> --description=%country% --mapid=%FID%0000 >> c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf >> would using the precomp-sea parameter for splitter improve >> results? It's not that tiles with loads of sea actually failed. >> >> On 03.05.2014 09:00, Gerd Petermann wrote: >> >> Hi all, >> >> I've coded a quick hack which seems to improve the ratios. >> On the other hand, I don't see these large differences >> between >> smallest and largest img file. >> What part of the world should I try? >> Do you use the precomp-sea parameter in splitter? >> >> Gerd >> >> >> ------------------------------------------------------------------------ >> Date: Wed, 30 Apr 2014 14:59:32 +0200 >> From: > extremecarver@ > <mailto: > extremecarver@ > > >> To: > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> if that doesn't seriously (more than 30-40%) slow down the >> splitter, I assume it would be much better... >> On 30.04.2014 14:06, Gerd Petermann wrote: >> >> Hi, >> >> if I got that right the number of nodes is not >> highly correlated to the img size, so the max-nodes >> value is not a good estimate. >> >> I assume the reason is that nodes which belong to >> roads produce a lot more bytes >> in the img file compared to nodes which are parts >> of shapes or other non-routable ways, not talking >> about nodes which are simply ignored by the style. >> >> So, a possible solution in splitter could be to parse >> the ways before reading the nodes and save all nodeids >> which belong to ways with highway=*. >> If these nodes are refered by more than one way with >> highway=* >> we assume that they will be routing nodes. >> These special nodes could be counted e.g. 10 times to >> give a better estimate. >> >> Gerd >> >> > Date: Wed, 30 Apr 2014 13:36:19 +0200 >> > From: > osm@ > <mailto: > osm@ > > >> > To: > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > Subject: Re: [mkgmap-dev] mkgmap ToDo list >> > >> > Multithreading the tile rendering for a single map >> is indeed a bit >> > difficult and I gave it up because you need to keep >> track which image >> > id's are already in use. Since I provide multiple >> maps the work-around >> > is running a few scripts parallel, which is also a >> crude form of >> > multithreading. >> > >> > The script language is PHP and it doesn't run on >> Windows without some >> > changes ('/' vs '\' in paths, 'rm -rf', that sort of >> stuff). Never tried it. >> > >> > To get a better optimum in file size, using the >> process I described >> > earlier, you could start off with a huge --max-nodes >> setting and then >> > 'search' for the highest --max-nodes that works for >> each specific area. >> > >> > On 30/04/2014 11:49, Felix Hartmann wrote: >> > > I would love if there was a possibility that you >> pass the used max-nodes >> > > value to mkgmap. >> > > When mkgmap is compiling the maps, then after the >> .img is created it >> > > should check >> > > a) did it crash due to too many max-nodes >> > > b) for me not important - but for others with very >> old GPS, etrex 10, >> > > ---> is tile bigger than X (usually 8) MB. >> > > >> > > if a) or b) true, then pass the file back to >> splitter and split with 60% >> > > of maxnodes - and compile the resulting .img files >> again. Should it fail >> > > again, use 40%, again 25%... Sometimes there are >> awful tiles, that need >> > > supersmall max-nodes till they compile, however >> lately (last 1-2 years) >> > > I never encountered them anymore. I think that >> happened rather due to a >> > > but in splitter/mgkmap that is fixed now. >> > > >> > > okay, you could also do this with a script, but it >> gets rather >> > > complicated to multithread it (you need to wait >> till mgkmap finished >> > > compiling all .img files - and run mkgmap first >> without address index to >> > > save time) and do some clever routines on making >> sure that the map id >> > > (e.g. 6340????.img) stay correct. Even more >> complicated to have >> > > consequent map id... >> > > >> > > For europe with a fixed max-node I get tiles from >> 1.9MB up to 18MB. >> > > That's factor 9 - so it's huge... >> > > If I could narrows that down easily to 8-18MB - >> without getting tiles >> > > crashing due to too high max-nodes values, that >> would be sweet. >> > > >> > > >> > > >> > > As for the scripts - would they run on windows >> too? - What programming >> > > language are they in? >> > > >> > > >> > > On 29.04.2014 21:39, > osm@ >> <mailto: > osm@ > > wrote: >> > >> Oh, and ofcourse anyone interested can get my >> scripts, send an email. >> > >> They'll be on Github someday anyway. >> > >> >> > >> >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: >> > >>> Hi Lambertus, >> > >>> >> > >>> okay, if I got that right you finally get *.img >> files with a size >> > >>> near (but below) 8MB, so maybe Henning can use >> that script, too. >> > >>> >> > >>> If you do that for e.g. Germany, how small is >> tpically the smallest >> > >>> *.img file ? >> > >>> Is it probably near 4 MB? >> > >>> >> > >>> Gerd >> > >>> >> > >>>> To: > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 >> > >>>> From: > osm@ > <mailto: > osm@ > > >> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> > >>>> >> > >>>> These are the direct results from Splitter. The >> format is o5m, both >> > >>>> input as output. Splitter version is: r321. >> > >>>> >> > >>>> For this test I split the original source with >> --max-nodes=8000000. >> > >>> Then >> > >>>> I render the initial tiles, when the result is >> larger than 8MB it's >> > >>>> subsplit again with >> --max-nodes=(8000000-(attempt*100000)). The >> > >>> initial >> > >>>> source files are ~70MB (o5m) and after several >> subsplits the two >> > >>> *.img >> > >>>> are < 8MB. During this process --max-nodes has >> been reduced to e.g. >> > >>>> 7500000 and the source file is split up in two >> o5m files of about >> > >>> 37MB. >> > >>>> >> > >>>> I can upload an example source file and it's >> two subsplit siblings >> > >>> if >> > >>>> you want. >> > >>>> >> > >>>> >> > >>>> On 2014-04-29 19:38, GerdP wrote: >> > >>>> > Hi Lambertus, >> > >>>> > >> > >>>> > that's interesting. Are these the img file >> sizes or the osm file >> > >>> sizes? >> > >>>> > >> > >>>> > Gerd >> > >>>> > >> > >>>> > >> > >>>> > Lambertus wrote >> > >>>> >> Unfortunately I cannot confirm that. Below >> is a bit of logging >> > >>> from my >> > >>>> >> script: >> > >>>> >> Original: 97000020 (70551453), New: 0 >> (35684445), New: 1 >> > >>> (36852845) >> > >>>> >> Original: 97000001 (74621042), New: 0 >> (37522992), New: 1 >> > >>> (37222739) >> > >>>> >> Original: 97000002 (73391358), New: 0 >> (37679505), New: 1 >> > >>> (38098627) >> > >>>> >> Original: 97000003 (77862567), New: 0 >> (39075311), New: 1 >> > >>> (39261197) >> > >>>> >> >> > >>>> >> The original files above contain contour >> data, the filesize is >> > >>> between >> > >>>> >> brackets. As you can see both resulting file >> are approximately >> > >>> the >> > >>>> >> same >> > >>>> >> size. >> > >>>> >> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: >> > >>>> >>> Hi Lambertus, >> > >>>> >>> >> > >>>> >>> and I guess that even after this >> optimization you will >> > >>>> >>> see a factor 3 or higher between the >> largest tile and the >> > >>> smallest. >> > >>>> >>> Can you confirm that? >> > >>>> >>> >> > >>>> >>> Gerd >> > >>>> >>> >> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 >> > >>>> >>>> From: >> > >>>> > >> > >>>> >> osm@ >> > >>>> > >> > >>>> >>>> To: >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> > >>>> >>>> >> > >>>> >>>> Num-tiles=x would indeed be better for >> this specific need. >> > >>>> >>>> >> > >>>> >>>> It is my experience that it regularly >> takes multiple calls to >> > >>>> >>> Splitter >> > >>>> >>>> to get 2+ sub-tiles when you reduce the >> max-nodes by 100k for >> > >>> each >> > >>>> >>>> sub-split attempt. This is what I >> currently do to get an >> > >>> optimum in >> > >>>> >>>> tile-size vs total number of tiles. >> > >>>> >>>> >> > >>>> >>>> >> > >>>> >>>> >> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: >> > >>>> >>>> > Hi Lambertus, >> > >>>> >>>> > >> > >>>> >>>> > that sounds like a possible change in >> splitter: >> > >>>> >>>> > Instead of specifying max-nodes you may >> specify --num-tiles=x >> > >>>> >>>> > and splitter will try to find a split >> that produces excactly >> > >>> x >> > >>>> >>> tiles >> > >>>> >>>> > which are not too narrow and have a node >> number which is not >> > >>>> >>>> > too far from the average (but still >> aligned to a multiple of >> > >>> map >> > >>>> >>> units >> > >>>> >>>> > as now). >> > >>>> >>>> > So, for your script that means you don't >> have to find the >> > >>>> >>> max-nodes >> > >>>> >>>> > value. >> > >>>> >>>> > >> > >>>> >>>> > I'll think about this again... >> > >>>> >>>> > >> > >>>> >>>> > Gerd >> > >>>> >>>> > >> > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >> > >>>> >>>> > > From: >> > >>>> > >> > >>>> >> osm@ >> > >>>> > >> > >>>> >>>> > > To: >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo >> list >> > >>>> >>>> > > >> > >>>> >>>> > > While this possibly can be solved in >> Splitter or Mkgmap, it >> > >>>> >>> could also >> > >>>> >>>> > > be solved by your build-script when >> you add a maximum tile >> > >>> size >> > >>>> >>> check >> > >>>> >>>> > > and re-split (with a lower number of >> max-nodes) until you >> > >>> get >> > >>>> >>> two or >> > >>>> >>>> > > more sub-tiles. Granted, this adds >> complexity to the script >> > >>> but >> > >>>> >>> it works >> > >>>> >>>> > > well for me. >> > >>>> >>>> > > >> > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland >> wrote: >> > >>>> >>>> > > > Hi Gerd, >> > >>>> >>>> > > > >> > >>>> >>>> > > > I would like to have img-tiles which >> have globally nearly >> > >>> the >> > >>>> >>> same >> > >>>> >>>> > > > filesize, so that they use the space >> of devices like >> > >>> eTrex 10. >> > >>>> >>>> > > > >> > >>>> >>>> > > > With my actual map I use globally >> the same value for >> > >>>> >>> max-nodes. But the >> > >>>> >>>> > > > size of the img-tiles differ more >> then factor 2. Eg. a >> > >>> tile in >> > >>>> >>> Germany >> > >>>> >>>> > > > is between 2 and 5 mb where a tile >> in China is about 10 >> > >>> mb. If >> > >>>> >>> I remove >> > >>>> >>>> > > > details, this difference will >> increase, because in >> > >>> Germany >> > >>>> >>> more objects >> > >>>> >>>> > > > will be removed from the img-tile >> then in China. >> > >>>> >>>> > > > >> > >>>> >>>> > > > Henning >> > >>>> >>>> > > > >> > >>>> >>>> > > > >> _______________________________________________ >> > >>>> >>>> > > > mkgmap-dev mailing list >> > >>>> >>>> > > > >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> > > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> >>>> > > >> > >>>> >>>> > > >> _______________________________________________ >> > >>>> >>>> > > mkgmap-dev mailing list >> > >>>> >>>> > > >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> _______________________________________________ >> > >>>> >>>> > mkgmap-dev mailing list >> > >>>> >>>> > >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> >>>> > >> > >>>> >>>> >> > >>>> >>>> >> _______________________________________________ >> > >>>> >>>> mkgmap-dev mailing list >> > >>>> >>>> >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> >>> >> > >>>> >>> >> _______________________________________________ >> > >>>> >>> mkgmap-dev mailing list >> > >>>> >>> >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> >> _______________________________________________ >> > >>>> >> mkgmap-dev mailing list >> > >>>> > >> > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> >> > >>>> > >> > >>>> >> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > -- >> > >>>> > View this message in context: >> > >>>> > >> > >>> >> >> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html >> > >>>> > Sent from the Mkgmap Development mailing list >> archive at >> > >>> Nabble.com. >> > >>>> > _______________________________________________ >> > >>>> > mkgmap-dev mailing list >> > >>>> > > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > >>>> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>>> _______________________________________________ >> > >>>> mkgmap-dev mailing list >> > >>>> > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >>> >> > >>> _______________________________________________ >> > >>> mkgmap-dev mailing list >> > >>> > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > >>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >> _______________________________________________ >> > >> mkgmap-dev mailing list >> > >> > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > > >> > >> > _______________________________________________ >> > mkgmap-dev mailing list >> > > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> > mkgmap-dev at .org > <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> -- >> keep on biking and discovering new trails >> >> Felix >> openmtbmap.org &www.velomap.org >> <http://www.velomap.org> >> >> >> _______________________________________________ mkgmap-dev >> mailing list > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> > mkgmap-dev at .org > <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> -- >> keep on biking and discovering new trails >> >> Felix >> openmtbmap.org &www.velomap.org <http://www.velomap.org> >> >> >> _______________________________________________ mkgmap-dev >> mailing list > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> > mkgmap-dev at .org > <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> >> _______________________________________________ mkgmap-dev mailing >> list > mkgmap-dev at .org >> <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> > mkgmap-dev at .org > <mailto: > mkgmap-dev at .org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> >> _______________________________________________ mkgmap-dev mailing >> list > mkgmap-dev at .org > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> > mkgmap-dev at .org >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org & www.velomap.org > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at .org > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805221.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
- Previous message: [mkgmap-dev] mkgmap ToDo list
- Next message: [mkgmap-dev] mkgmap ToDo list
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list