logo separator

[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

From Carlos Dávila carlos at alternativaslibres.org on Sun Sep 12 20:34:05 BST 2021

I share the same issue exposed by Felix, but not only for contour lines 
but also for DEM, which I have precompiled as *.img for many countries 
and just combine with the updated OSM tiles.

El 12/9/21 a las 19:10, Felix Hartmann escribió:
> And well - it is burning SSD for the contourlines currently even if 
> not calling multiple times. 130GB minimum per worldwide map update for 
> all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling 
> multiple times and 10m and 20m I had about 2TB of useless writes per 
> weekly map update. I've got rid of all of them with my uncommenting 
> the line, plus saved about 500GB of writes by now doing all the 
> gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk 
> writes per map update I'm down to 1TB.. (and yeah the resplitting of 
> tiles added another 5TB of writes - but that could have been fixed 
> easily by changing order of commands too).
> On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com 
> <mailto:extremecarver at gmail.com>> wrote:
>     because you need to do it if you want to show contourlines. Now
>     compiling the worldwide contourlines each time again - would burn
>     SSD as well - besides taking longer than just compiling the maps.
>     So everyone who offers maps for many countries as download puts
>     the contourlines separate. If you want to offer the choice of
>     showing contourlines or not - mkmgap will write the gmap
>     contourlines once not needed, and once the maps not needed. If you
>     don't want to offer that option - it's only the contourlines being
>     written without need. Contourlines for all geofabrik extracts 20m
>     equidistance are about 130GB, 10m would be 260GB.
>     If you offer 10m and 20m it will be even more not needed writes.
>     The only solution is to go for a Ramdisk instead - However you
>     likely will need 128GB of RAM to do that - because for Asia or
>     Europe you need a 64GB Ramdisk. Same for North America extract
>     (but few people offer that and just have Canada and the US divided
>     into the 4-5 zones). For other maps you will get by with a 15-20GB
>     Ramdisk (which I have resorted to now for all but the windows
>     installers because I don't want to let the server wait for ages
>     for the single thread NSIS wrapping up the data and instead start
>     the next country in parallel). And yes going RAMDISK already using
>     my patch and symlinking those files so mkgmap cannot overwrite
>     them (would still be faster if mkgmap didn't try in first place I
>     think). If you want to write out the contourlines as well besides
>     the additional time - for Asia continent you may then go for a
>     90GB Ramdisk minimum if offering windows and gmap installers. In
>     this case gmapsupp.img donwnloads aren't possible anyhow due to
>     the huge data amounts. For smaller countries I have them too....
>     I coded around this now by using symlinks - but yeah that will be
>     quite a lot of work and is prone to break - while I guess it's
>     only 10 lines or so to add to mkgmap code to have a mode that does
>     not write them out if they are present - or if you tell on
>     commandline they are present already. For .img files they aren't
>     overwritten either...
>     On Sun, 12 Sept 2021 at 18:40, Gerd Petermann
>     <gpetermann_muenchen at hotmail.com
>     <mailto:gpetermann_muenchen at hotmail.com>> wrote:
>         Hi Felix,
>         did not read all the posts in detail. I understand that mkgmap
>         is neither burning SSDs nor doing any excessive writing unless
>         you call it multiple times for the same input files. So the
>         question is: Why do you do that?
>         Gerd
>         ________________________________________
>         Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag
>         von Felix Hartmann <extremecarver at gmail.com
>         <mailto:extremecarver at gmail.com>>
>         Gesendet: Freitag, 10. September 2021 00:02
>         An: Development list for mkgmap
>         Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and
>         converting to disk if used with --gmapi option
>         Well I think the .tmp files are just building up - and the
>         renamed. So they are not causing any actual excessive write.
>         As for the gmap - it would be cool if there is a mode to not
>         write them.
>         Actually it would be great if mkgmap could write all in one
>         go. Because the thing that takes so much time - is the address
>         search - and that is always the same. The differences are tiny
>         (just because MapInstall is crashing when files are missing)
>         you need to compile them separately.
>         but maybe there could be a mode where mkgamp writes all in one go.
>         So family-name / family-name1 / family-name2
>         description / description1 / description2
>         input input1 input2
>         family-name..
>         show-profiles
>         overview-mapname
>         product-id
>         (and maybe I missed some options are those that would need to
>         be given for each set of input tiles. And then just an option
>         where you tell mkgmap files starting with which first 4
>         numbers are relevant for address search. No need to analyze if
>         those other supplied .img (e.g. buildings or contourlines)
>         need to be added to address search.). I know coded around the
>         problem of the gmap files causing excessive writes. But yeah
>         that is actually really complicated be it on windows or linux...
>         On Wed, 8 Sept 2021 at 17:07, Felix Hartmann
>         <extremecarver at gmail.com
>         <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com
>         <mailto:extremecarver at gmail.com>>> wrote:
>         Well I could give it 20 GB ram disk, maybe 32 but then I need
>         to render on less than 12 processes 64GB ram available). But
>         that is not enough for Asia continent map, and I guess super
>         tight for Europe...
>         Mkgmap could definitely keep those .tmp files in memory. But
>         the important bit is the gmap files not needeed to be
>         written.... Also would save quite some CPU time.
>         On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org
>         <mailto:witwall3 at disroot.org><mailto:witwall3 at disroot.org
>         <mailto:witwall3 at disroot.org>>> wrote:
>         On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
>         >
>         > Yes on an nvme disk you barely notice the conversion - it's
>         really quick.
>         > BUT it is not needed if you have the files and even more -
>         it burns your
>         > NVME SSD disk.
>         +1. I always use an old spinning rust disk when using mkgmap
>         to save ssd
>         write cycles, even without contours and such. It seems to be
>         profligate
>         in its use of disk cycles. I did try using RAM disk, but even
>         with 16GB
>         on a laptop, that was soon exhausted.
>         ael
>         _______________________________________________
>         mkgmap-dev mailing list
>         mkgmap-dev at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev at lists.mkgmap.org.uk>>
>         https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>         <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
>         --
>         Felix Hartman - Openmtbmap.org & VeloMap.org
>         _______________________________________________
>         mkgmap-dev mailing list
>         mkgmap-dev at lists.mkgmap.org.uk
>         <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>         https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>         <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
>     -- 
>     Felix Hartman - Openmtbmap.org & VeloMap.org
> -- 
> Felix Hartman - Openmtbmap.org & VeloMap.org
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

More information about the mkgmap-dev mailing list