[mkgmap-dev] splitter r325: improved split algo and new option
From Gerd Petermann gpetermann_muenchen at hotmail.com on Thu May 8 09:01:13 BST 2014
Hi Lambertus, maybe also look at https://wiki.openstreetmap.org/wiki/Mkgmap/help/splitter#Tuning (the page needs some updates, but that part is correct) Gerd > Date: Thu, 8 May 2014 09:31:28 +0200 > From: osm at na1400.info > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option > > Thanks Gerd! This is valuable information for those of us processing > large areas of the planet. > > Unfortunately there is no additional speedup for me because I already > use o5m because of osmupdate (to keep a local planet copy up-to-date). > > On 07/05/2014 11:59, Gerd Petermann wrote: > > Hi Felix, > > > > try o5m for both input and output, it is much faster. > > The command > > osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m > > runs quite fast (~70 seconds on my machine), > > the o5m file is ~2.430 MB, the pbf file has 2.104 MB. > > > > Splitter is much faster reading from o5m when > > the keep-complete option is in use. > > (210 secs for the o5m, 441 for pbf) > > > > With --output=pbf both are slower, and mkgmap is also much slower. > > > > All times with I/O on one normal hard disk. Even better results if you have > > two different disks for reading and writing. > > > > Gerd > > > > ------------------------------------------------------------------------ > > Date: Wed, 7 May 2014 11:37:58 +0200 > > From: extremecarver at gmail.com > > To: mkgmap-dev at lists.mkgmap.org.uk > > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option > > > > Well I still use pbf and not o5m. > > First pbf is smaller.. > > Second - Geofabrik only offers pbf - that's why I stayed with it. > > > > I don't think I can cut a lot of time by first converting to 05m, then > > hand it over to splitter... > > Actually I also let splitter output pbf... Maybe I could change that in > > future to 05m.. > > On 07.05.2014 11:36, Gerd Petermann wrote: > > > > Hi Felix, > > > > well, nowadays splitter performance mostly depends on I/O if you use > > o5m format > > for input and output and give enough heap. > > > > Reg. mkgmap performance improvements: yes, that's what I expected. > > In short, the branch improved the evaluation of tags and the > > creation of the NOD file. > > > > Gerd > > > > > > ------------------------------------------------------------------------ > > Date: Wed, 7 May 2014 11:29:10 +0200 > > From: extremecarver at gmail.com <mailto:extremecarver at gmail.com> > > To: mkgmap-dev at lists.mkgmap.org.uk > > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new > > option > > > > Well - I'll update all my maps on Thursday again, to recheck. Maybe > > it has to do with increasing-maxnodes? Though I thought the higher > > the max-nodes, the faster... > > And I only meant splitter. I upgraded mkgmap at the same time (now > > integrating performance branch changes) - so mkgmap by itself got > > faster (though it depends on the country - seems like well mapped > > countries profit a lot more (e.g. Austria like 30% time off), than > > countries where few continue commands will be in action cause their > > mapping is basic like Asia). > > > > I'm not using any pre-split files or cached files of any sort either... > > On 07.05.2014 06:49, Gerd Petermann wrote: > > > > Hi Felix, > > > > reg. speed: I can't reproduce that. I compared a split of Germany, > > both versions (r321 and r325) are more or less running the same > > time. > > (I've executed both programs two times to make sure that disk > > caches > > are not causing big differences) > > > > Or did you mean the combination of splitter + mkgmap to process > > e.g. Asia? > > > > Gerd > > > > ------------------------------------------------------------------------ > > Date: Tue, 6 May 2014 18:22:00 +0200 > > From: extremecarver at gmail.com <mailto:extremecarver at gmail.com> > > To: mkgmap-dev at lists.mkgmap.org.uk > > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and > > new option > > > > Seems to be much better now. I don't think I can increase the > > max-nodes value though, but for most maps the new algo creates > > less tiles for the same max-nodes value (e.g. Austria from 43 > > down to 35 for me, with the smallest tile now around 5MB instead > > of 2.8, and the biggest 12MB instead of 11MB, for Asia I > > simultaneously increased max-nodes from 800k to 900k- so I'm > > down from 624 tiles to 493.... and size from 970KB-16MB to now > > ). So it still seems to depend on the country, but it's already > > a lot better... > > It's a bit slower (about 10% more time) > > > > On 06.05.2014 13:56, Gerd Petermann wrote: > > > > Hi all, > > > > I've applied num-tiles-v1.patch and improved the split > > algo, see > > http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html > > > > > > It is now less likely that splitter creates tiles with a low > > number of > > nodes, it is more likely that all tiles have nearly the same > > number of nodes, > > and typically you will see fewer tiles. > > Maybe this also means that you can increase the max-nodes value. > > > > I hope this also reduces the need for complex interactions > > between > > spltter and mkgmap. > > > > > > > > Gerd > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > keep on biking and discovering new trails > > > > Felix > > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > > > > _______________________________________________ mkgmap-dev > > mailing list mkgmap-dev at lists.mkgmap.org.uk > > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > keep on biking and discovering new trails > > > > Felix > > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > > > > _______________________________________________ mkgmap-dev mailing > > list mkgmap-dev at lists.mkgmap.org.uk > > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > keep on biking and discovering new trails > > > > Felix > > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > > > > _______________________________________________ mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140508/ee2a74cb/attachment-0001.html>
- Previous message: [mkgmap-dev] splitter r325: improved split algo and new option
- Next message: [mkgmap-dev] splitter r325: improved split algo and new option
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list