logo separator

[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

From Felix Hartmann extremecarver at gmail.com on Sun Sep 12 18:04:14 BST 2021

because you need to do it if you want to show contourlines. Now compiling
the worldwide contourlines each time again - would burn SSD as well -
besides taking longer than just compiling the maps. So everyone who offers
maps for many countries as download puts the contourlines separate. If you
want to offer the choice of showing contourlines or not - mkmgap will write
the gmap contourlines once not needed, and once the maps not needed. If you
don't want to offer that option - it's only the contourlines being written
without need. Contourlines for all geofabrik extracts 20m equidistance are
about 130GB, 10m would be 260GB.
If you offer 10m and 20m it will be even more not needed writes.

The only solution is to go for a Ramdisk instead - However you likely will
need 128GB of RAM to do that - because for Asia or Europe you need a 64GB
Ramdisk. Same for North America extract (but few people offer that and just
have Canada and the US divided into the 4-5 zones). For other maps you will
get by with a 15-20GB Ramdisk (which I have resorted to now for all but the
windows installers because I don't want to let the server wait for ages for
the single thread NSIS wrapping up the data and instead start the next
country in parallel). And yes going RAMDISK already using my patch and
symlinking those files so mkgmap cannot overwrite them (would still be
faster if mkgmap didn't try in first place I think). If you want to write
out the contourlines as well besides the additional time - for Asia
continent you may then go for a 90GB Ramdisk minimum if offering windows
and gmap installers. In this case gmapsupp.img donwnloads aren't possible
anyhow due to the huge data amounts. For smaller countries I have them

I coded around this now by using symlinks - but yeah that will be quite a
lot of work and is prone to break - while I guess it's only 10 lines or so
to add to mkgmap code to have a mode that does not write them out if they
are present - or if you tell on commandline they are present already. For
.img files they aren't overwritten either...

On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <
gpetermann_muenchen at hotmail.com> wrote:

> Hi Felix,
> did not read all the posts in detail. I understand that mkgmap is neither
> burning SSDs nor doing any excessive writing unless you call it multiple
> times for the same input files. So the question is: Why do you do that?
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
> Felix Hartmann <extremecarver at gmail.com>
> Gesendet: Freitag, 10. September 2021 00:02
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
> Well I think the .tmp files are just building up - and the renamed. So
> they are not causing any actual excessive write.
> As for the gmap - it would be cool if there is a mode to not write them.
> Actually it would be great if mkgmap could write all in one go. Because
> the thing that takes so much time - is the address search - and that is
> always the same. The differences are tiny (just because MapInstall is
> crashing when files are missing) you need to compile them separately.
> but maybe there could be a mode where mkgamp writes all in one go.
> So family-name / family-name1 / family-name2
> description / description1 / description2
> input input1 input2
> family-name..
> show-profiles
> overview-mapname
> product-id
> (and maybe I missed some options are those that would need to be given for
> each set of input tiles. And then just an option where you tell mkgmap
> files starting with which first 4 numbers are relevant for address search.
> No need to analyze if those other supplied .img (e.g. buildings or
> contourlines) need to be added to address search.). I know coded around the
> problem of the gmap files causing excessive writes. But yeah that is
> actually really complicated be it on windows or linux...
> On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> Well I could give it 20 GB ram disk, maybe 32 but then I need to render on
> less than 12 processes 64GB ram available). But that is not enough for Asia
> continent map, and I guess super tight for Europe...
> Mkgmap could definitely keep those .tmp files in memory. But the important
> bit is the gmap files not needeed to be written.... Also would save quite
> some CPU time.
> On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org<mailto:
> witwall3 at disroot.org>> wrote:
> On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
> >
> > Yes on an nvme disk you barely notice the conversion - it's really quick.
> > BUT it is not needed if you have the files and even more - it burns your
> > NVME SSD disk.
> +1. I always use an old spinning rust disk when using mkgmap to save ssd
> write cycles, even without contours and such. It seems to be profligate
> in its use of disk cycles. I did try using RAM disk, but even with 16GB
> on a laptop, that was soon exhausted.
> ael
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210912/e0a68039/attachment.html>

More information about the mkgmap-dev mailing list