[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Mon Sep 13 11:06:20 BST 2021

Not sure if it makes it possible to use read only attribute instead of
moving and mklink. Maybe yes - because that was not possible before. So it
then would be set read only - instead of of move & mklink - and at the end
remove read only instead of moving back.

On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver at gmail.com>
wrote:

> Thanks Gerd,
>
> but that is just removing the warning if it cannot overwrite, correct?
> If it can overwrite the gmap file it will still overwrite it as I see.. So
> in essence just a bit more intelligent then my disabling that line.
>
> I think it should not overwrite at all if present and --x-gmapi-minimal
> (then you don't have to move away the files and link them back to the
> original folder).
>
> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
> gpetermann_muenchen at hotmail.com> wrote:
>
>> Hi all,
>>
>> attached is a quick patch that implements experimaltal option
>> --x-gmapi-minimal.
>>
>> If used instead of --gmapi mkgmap will not fail if a write-protected
>> output file exists in the gmapi output folder and mkgmap copies data from
>> *.img. It should still crash when other files like Info.xml are written.
>>
>> BTW: no conversion is done when those files are written. I think mkgmap
>> simply copies data from sub files in *.img into single files. So, the same
>> sequence of Bytes exists two or more times on the Computer. Windows Server
>> seems to support automatic data deduplication (1). I am sure Linux offers
>> similar support. No idea how effective or reliable it is, but it might be
>> worth trying.
>>
>> Gerd
>> (1)
>>
>> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
>> Carlos Dávila <carlos at alternativaslibres.org>
>> Gesendet: Sonntag, 12. September 2021 22:45
>> An: mkgmap-dev at lists.mkgmap.org.uk
>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>> to disk if used with --gmapi option
>>
>> I think it's a good idea if mkgmap checks the required files are present.
>>
>> El 12/9/21 a las 21:16, Gerd Petermann escribió:
>> > Hi Felix,
>> >
>> > so you just want a new option so that mkgmap doesn't fail if it cannot
>> overwrite (write-protected) files in the output directory, right? No need
>> to verify the content of the exiting file(s)?
>> >
>> > Gerd
>> >
>> > ________________________________________
>> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
>> von Felix Hartmann <extremecarver at gmail.com>
>> > Gesendet: Sonntag, 12. September 2021 19:10
>> > An: Development list for mkgmap
>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>> to disk if used with --gmapi option
>> >
>> > And well - it is burning SSD for the contourlines currently even if not
>> calling multiple times. 130GB minimum per worldwide map update for all
>> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
>> times and 10m and 20m I had about 2TB of useless writes per weekly map
>> update. I've got rid of all of them with my uncommenting the line, plus
>> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
>> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
>> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
>> writes - but that could have been fixed easily by changing order of
>> commands too).
>> >
>> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com
>> <mailto:extremecarver at gmail.com>> wrote:
>> > because you need to do it if you want to show contourlines. Now
>> compiling the worldwide contourlines each time again - would burn SSD as
>> well - besides taking longer than just compiling the maps. So everyone who
>> offers maps for many countries as download puts the contourlines separate.
>> If you want to offer the choice of showing contourlines or not - mkmgap
>> will write the gmap contourlines once not needed, and once the maps not
>> needed. If you don't want to offer that option - it's only the contourlines
>> being written without need. Contourlines for all geofabrik extracts 20m
>> equidistance are about 130GB, 10m would be 260GB.
>> > If you offer 10m and 20m it will be even more not needed writes.
>> >
>> > The only solution is to go for a Ramdisk instead - However you likely
>> will need 128GB of RAM to do that - because for Asia or Europe you need a
>> 64GB Ramdisk. Same for North America extract (but few people offer that and
>> just have Canada and the US divided into the 4-5 zones). For other maps you
>> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
>> but the windows installers because I don't want to let the server wait for
>> ages for the single thread NSIS wrapping up the data and instead start the
>> next country in parallel). And yes going RAMDISK already using my patch and
>> symlinking those files so mkgmap cannot overwrite them (would still be
>> faster if mkgmap didn't try in first place I think). If you want to write
>> out the contourlines as well besides the additional time - for Asia
>> continent you may then go for a 90GB Ramdisk minimum if offering windows
>> and gmap installers. In this case gmapsupp.img donwnloads aren't possible
>> anyhow due to the huge data amounts. For sm
>> >   aller countries I have them too....
>> >
>> > I coded around this now by using symlinks - but yeah that will be quite
>> a lot of work and is prone to break - while I guess it's only 10 lines or
>> so to add to mkgmap code to have a mode that does not write them out if
>> they are present - or if you tell on commandline they are present already.
>> For .img files they aren't overwritten either...
>> >
>> > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <
>> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
>> wrote:
>> > Hi Felix,
>> >
>> > did not read all the posts in detail. I understand that mkgmap is
>> neither burning SSDs nor doing any excessive writing unless you call it
>> multiple times for the same input files. So the question is: Why do you do
>> that?
>> >
>> > Gerd
>> >
>> > ________________________________________
>> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
>> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
>> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
>> > Gesendet: Freitag, 10. September 2021 00:02
>> > An: Development list for mkgmap
>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>> to disk if used with --gmapi option
>> >
>> > Well I think the .tmp files are just building up - and the renamed. So
>> they are not causing any actual excessive write.
>> > As for the gmap - it would be cool if there is a mode to not write them.
>> >
>> > Actually it would be great if mkgmap could write all in one go. Because
>> the thing that takes so much time - is the address search - and that is
>> always the same. The differences are tiny (just because MapInstall is
>> crashing when files are missing) you need to compile them separately.
>> >
>> > but maybe there could be a mode where mkgamp writes all in one go.
>> > So family-name / family-name1 / family-name2
>> > description / description1 / description2
>> > input input1 input2
>> > family-name..
>> > show-profiles
>> > overview-mapname
>> > product-id
>> > (and maybe I missed some options are those that would need to be given
>> for each set of input tiles. And then just an option where you tell mkgmap
>> files starting with which first 4 numbers are relevant for address search.
>> No need to analyze if those other supplied .img (e.g. buildings or
>> contourlines) need to be added to address search.). I know coded around the
>> problem of the gmap files causing excessive writes. But yeah that is
>> actually really complicated be it on windows or linux...
>> >
>> > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver at gmail.com
>> <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:
>> extremecarver at gmail.com>>> wrote:
>> > Well I could give it 20 GB ram disk, maybe 32 but then I need to render
>> on less than 12 processes 64GB ram available). But that is not enough for
>> Asia continent map, and I guess super tight for Europe...
>> >
>> > Mkgmap could definitely keep those .tmp files in memory. But the
>> important bit is the gmap files not needeed to be written.... Also would
>> save quite some CPU time.
>> >
>> > On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org<mailto:
>> witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto:
>> witwall3 at disroot.org>>> wrote:
>> > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
>> >> Yes on an nvme disk you barely notice the conversion - it's really
>> quick.
>> >> BUT it is not needed if you have the files and even more - it burns
>> your
>> >> NVME SSD disk.
>> > +1. I always use an old spinning rust disk when using mkgmap to save ssd
>> > write cycles, even without contours and such. It seems to be profligate
>> > in its use of disk cycles. I did try using RAM disk, but even with 16GB
>> > on a laptop, that was soon exhausted.
>> >
>> > ael
>> >
>> >
>> > _______________________________________________
>> > mkgmap-dev mailing list
>> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
>> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
>> mkgmap-dev at lists.mkgmap.org.uk>>
>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> >
>> >
>> > --
>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>> >
>> > _______________________________________________
>> > mkgmap-dev mailing list
>> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> >
>> >
>> > --
>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>> >
>> >
>> >
>> > --
>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>> >
>> > _______________________________________________
>> > mkgmap-dev mailing list
>> > mkgmap-dev at lists.mkgmap.org.uk
>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210913/897047d6/attachment-0001.html>