[mkgmap-dev] splitter r325: improved split algo and new option
From Gerd Petermann gpetermann_muenchen at hotmail.com on Wed May 7 15:43:46 BST 2014
Hi Felix, I seem to remember that the disadvantage of pbf for output of splitter was higher, but maybe that was before I modified splitter to use a smaller "BatchLimit" in the pbf write routine: configBatchLimit(1000); The default value produced slightly better compression but required much more heap. The original code was optimized for one write process, while splitter may start many hundrets, and in some situations splitter and pbf writer were fighting for the heap, causing heavy work for GC. Of course this only happened when splitting large files like Europe or planet. Gerd Date: Wed, 7 May 2014 16:06:27 +0200 From: extremecarver at gmail.com To: mkgmap-dev at lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Well, yes - I mainly tried on Austria now, so results may differ a bit by country - but for me the trend is clear. Starting with a .pbf file. Fastest is osmcovert it to o5m (about 15seconds) then splitter output to o5m. Total time to split 1:11 Second fastest os osmconvert it to o5m - then splitter output osm.pbf =1:14 Slowest is splitter osm.pbf to osm.pbf = 2:04min.. For mkgmap austria: osm.pbf (and creating mapset files twice): 3:32 o5m (and creating mapset files twice): 3:30... So for right now, I will output splitter to osm.pbf, but convert first to o5m with osmconvert. After successful split, I directly delete the o5m file. This is using a conventional hdd, maybe with an SSD the o5m advantage is a bit bigger (but then you might be more space bound on the SSD too) The output to osm.pbf by splitter is simply more effective - because o5m is 50% bigger. And for Austria the overall time it is faster is about 5 seconds... Of course osmconvert also needs some time (about 15seconds of the 1:11) - so If I started with an o5m file instead of osm.pbf - it would save that 15seconds... I only update my maps once a week (and europe continent only every ~6 weeks) - so even if I download the planet - I will have to see if I maybe cut it to osm.pbf instead of o5m files to save space... I don't have time right now to set up all that toolchain however. My server is on a 1000Mbit connection - so downloads are faster than the HDD if the source is also fast... For sure converting to o5m before feeding tiles to splitter - makes a lot of sense. Even if all else stays osm.pbf.. On 07.05.2014 15:28, Gerd Petermann wrote: Hi Felix, yes, confirmed. The output files are much bigger because splitter never writes the version info (timestamp,uid,user,changeset, version). In mkgmap we read the file only once, not multiple times as in splitter, so the advantage is rather small. Gerd > Date: Wed, 7 May 2014 15:20:51 +0200 > From: extremecarver at gmail.com > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option > > Well, I just ran splitter on the .osm.pbf file for Asia and austria. > True the split is a bit faster to o5m - but the difference is not so big. > > Compiling from osm.pbf or o5m however - was more or less the same speed... > on the other hand the o5m splitted files use much more space than > osm.pbf (for Austria 293MB vs 450MB - that's about 50% more). > > I will do some further test with input o5m and output osm.pbf vs output > o5m. However if there is also no significant time difference (right now > it's like maybe 10% faster output to o5m instead of osm.pbf) - then I > think I'm gonna stay splitter output at osm.pbf - simply because on my > server I'm a bit harddisk bound (got a large network drive for saving > stuff, but with about 20MB/s vs 150MB/s for the local drive, it's really > only useful to save stuff there - not for working with anything on the > network drive). Splitter input according to the above time measures > however - really seems to profit from o5m... > On 07.05.2014 12:56, Felix Hartmann wrote: > > okay, well the numbers are convincing. I will change to o5m and > > osmupdate too - as soon as I find the time (I do think I need 5-10 > > hours changing my scripts and making sure everything is neatly cut, > > but right now I'm too timelimited to do so).. > > On 07.05.2014 12:15, Bernd Weigelt wrote: > >> Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann: > >>> Well I still use pbf and not o5m. > >>> First pbf is smaller.. > >>> Second - Geofabrik only offers pbf - that's why I stayed with it. > >>> > >>> I don't think I can cut a lot of time by first converting to 05m, then > >>> hand it over to splitter... > >>> Actually I also let splitter output pbf... Maybe I could change that in > >>> future to 05m.. > >> I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the > >> needed > >> polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. > >> takes now 3 > >> minutes. I update the planet once a month, my extracts everytime i > >> need them. > >> > >> thats much faster then download a new pbf everyday or update this pbf. > >> and with the local planet it is easier to create special maps for > >> friends like > >> bonn+100 or dach+ > >> > >> Bernd > >> > >> > > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org & www.velomap.org > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140507/c2129445/attachment-0001.html>
- Previous message: [mkgmap-dev] splitter r325: improved split algo and new option
- Next message: [mkgmap-dev] splitter r325: improved split algo and new option
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list