[mkgmap-dev] splitter r325: improved split algo and new option

Thu May 8 08:31:28 BST 2014

Thanks Gerd! This is valuable information for those of us processing 
large areas of the planet.

Unfortunately there is no additional speedup for me because I already 
use o5m because of osmupdate (to keep a local planet copy up-to-date).

On 07/05/2014 11:59, Gerd Petermann wrote:
> Hi Felix,
>
> try o5m for both input and output, it is much faster.
> The command
> osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m
> runs quite fast (~70 seconds on my machine),
> the o5m file is ~2.430 MB, the pbf file has 2.104 MB.
>
> Splitter is much faster reading from o5m when
> the keep-complete option is in use.
> (210 secs for the o5m, 441 for pbf)
>
> With --output=pbf both are slower, and mkgmap is also much slower.
>
> All times with I/O on one normal hard disk. Even better results if you have
> two different disks for reading and writing.
>
> Gerd
>
> ------------------------------------------------------------------------
> Date: Wed, 7 May 2014 11:37:58 +0200
> From: extremecarver at gmail.com
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
>
> Well I still use pbf and not o5m.
> First pbf is smaller..
> Second - Geofabrik only offers pbf - that's why I stayed with it.
>
> I don't think I can cut a lot of time by first converting to 05m, then
> hand it over to splitter...
> Actually I also let splitter output pbf... Maybe I could change that in
> future to 05m..
> On 07.05.2014 11:36, Gerd Petermann wrote:
>
>     Hi Felix,
>
>     well, nowadays splitter performance mostly depends on I/O if you use
>     o5m format
>     for input and output and give enough heap.
>
>     Reg. mkgmap performance improvements: yes, that's what I expected.
>     In short, the branch improved the evaluation of tags and the
>     creation of the NOD file.
>
>     Gerd
>
>
>     ------------------------------------------------------------------------
>     Date: Wed, 7 May 2014 11:29:10 +0200
>     From: extremecarver at gmail.com <mailto:extremecarver at gmail.com>
>     To: mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new
>     option
>
>     Well - I'll update all my maps on Thursday again, to recheck. Maybe
>     it has to do with increasing-maxnodes? Though I thought the higher
>     the max-nodes, the faster...
>     And I only meant splitter. I upgraded mkgmap at the same time (now
>     integrating performance branch changes) - so mkgmap by itself got
>     faster (though it depends on the country - seems like well mapped
>     countries profit a lot more (e.g. Austria like 30% time off), than
>     countries where few continue commands will be in action cause their
>     mapping is basic like Asia).
>
>     I'm not using any pre-split files or cached files of any sort either...
>     On 07.05.2014 06:49, Gerd Petermann wrote:
>
>         Hi Felix,
>
>         reg. speed: I can't reproduce that. I compared a split of Germany,
>         both versions (r321 and r325) are more or less running the same
>         time.
>         (I've executed both programs two times to make sure that disk
>         caches
>         are not causing big differences)
>
>         Or did you mean the combination of splitter + mkgmap to process
>         e.g. Asia?
>
>         Gerd
>
>         ------------------------------------------------------------------------
>         Date: Tue, 6 May 2014 18:22:00 +0200
>         From: extremecarver at gmail.com <mailto:extremecarver at gmail.com>
>         To: mkgmap-dev at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>         Subject: Re: [mkgmap-dev] splitter r325: improved split algo and
>         new option
>
>         Seems to be much better now. I don't think I can increase the
>         max-nodes value though, but for most maps the new algo creates
>         less tiles for the same max-nodes value (e.g. Austria from 43
>         down to 35 for me, with the smallest tile now around 5MB instead
>         of 2.8, and the biggest 12MB instead of 11MB, for Asia I
>         simultaneously increased max-nodes from 800k to 900k- so I'm
>         down from 624 tiles to 493.... and size from 970KB-16MB to now
>         ). So it still seems to depend on the country, but it's already
>         a lot better...
>         It's a bit slower (about 10% more time)
>
>         On 06.05.2014 13:56, Gerd Petermann wrote:
>
>             Hi all,
>
>             I've applied num-tiles-v1.patch  and improved the split
>             algo, see
>             http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
>
>
>             It is now less likely that splitter creates tiles with a low
>             number of
>             nodes, it is more likely that all tiles have nearly the same
>             number of nodes,
>             and typically you will see fewer tiles.
>             Maybe this also means that you can increase the max-nodes value.
>
>             I hope this also reduces the need for complex interactions
>             between
>             spltter and mkgmap.
>
>
>
>             Gerd
>
>
>             _______________________________________________
>             mkgmap-dev mailing list
>             mkgmap-dev at lists.mkgmap.org.uk  <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>             http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>         --
>         keep on biking and discovering new trails
>
>         Felix
>         openmtbmap.org &www.velomap.org  <http://www.velomap.org>
>
>
>         _______________________________________________ mkgmap-dev
>         mailing list mkgmap-dev at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>         http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>         _______________________________________________
>         mkgmap-dev mailing list
>         mkgmap-dev at lists.mkgmap.org.uk  <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>         http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>     --
>     keep on biking and discovering new trails
>
>     Felix
>     openmtbmap.org &www.velomap.org  <http://www.velomap.org>
>
>
>     _______________________________________________ mkgmap-dev mailing
>     list mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>     _______________________________________________
>     mkgmap-dev mailing list
>     mkgmap-dev at lists.mkgmap.org.uk  <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> keep on biking and discovering new trails
>
> Felix
> openmtbmap.org &www.velomap.org  <http://www.velomap.org>
>
>
> _______________________________________________ mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>