[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Mon Sep 13 11:09:12 BST 2021

Oh - but data was certainly written - A rename will not show as data
written in both Task manager on Windows, as well as in the smart data (I'm
using Windows 10 pro not Windows server however - maybe that functionality
is limited to windows server?)

On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver at gmail.com>
wrote:

> Not sure if it makes it possible to use read only attribute instead of
> moving and mklink. Maybe yes - because that was not possible before. So it
> then would be set read only - instead of of move & mklink - and at the end
> remove read only instead of moving back.
>
> On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver at gmail.com>
> wrote:
>
>> Thanks Gerd,
>>
>> but that is just removing the warning if it cannot overwrite, correct?
>> If it can overwrite the gmap file it will still overwrite it as I see..
>> So in essence just a bit more intelligent then my disabling that line.
>>
>> I think it should not overwrite at all if present and --x-gmapi-minimal
>> (then you don't have to move away the files and link them back to the
>> original folder).
>>
>> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
>> gpetermann_muenchen at hotmail.com> wrote:
>>
>>> Hi all,
>>>
>>> attached is a quick patch that implements experimaltal option
>>> --x-gmapi-minimal.
>>>
>>> If used instead of --gmapi mkgmap will not fail if a write-protected
>>> output file exists in the gmapi output folder and mkgmap copies data from
>>> *.img. It should still crash when other files like Info.xml are written.
>>>
>>> BTW: no conversion is done when those files are written. I think mkgmap
>>> simply copies data from sub files in *.img into single files. So, the same
>>> sequence of Bytes exists two or more times on the Computer. Windows Server
>>> seems to support automatic data deduplication (1). I am sure Linux offers
>>> similar support. No idea how effective or reliable it is, but it might be
>>> worth trying.
>>>
>>> Gerd
>>> (1)
>>>
>>> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>>>
>>> ________________________________________
>>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
>>> Carlos Dávila <carlos at alternativaslibres.org>
>>> Gesendet: Sonntag, 12. September 2021 22:45
>>> An: mkgmap-dev at lists.mkgmap.org.uk
>>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>>> to disk if used with --gmapi option
>>>
>>> I think it's a good idea if mkgmap checks the required files are present.
>>>
>>> El 12/9/21 a las 21:16, Gerd Petermann escribió:
>>> > Hi Felix,
>>> >
>>> > so you just want a new option so that mkgmap doesn't fail if it cannot
>>> overwrite (write-protected) files in the output directory, right? No need
>>> to verify the content of the exiting file(s)?
>>> >
>>> > Gerd
>>> >
>>> > ________________________________________
>>> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
>>> von Felix Hartmann <extremecarver at gmail.com>
>>> > Gesendet: Sonntag, 12. September 2021 19:10
>>> > An: Development list for mkgmap
>>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and
>>> converting to disk if used with --gmapi option
>>> >
>>> > And well - it is burning SSD for the contourlines currently even if
>>> not calling multiple times. 130GB minimum per worldwide map update for all
>>> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
>>> times and 10m and 20m I had about 2TB of useless writes per weekly map
>>> update. I've got rid of all of them with my uncommenting the line, plus
>>> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
>>> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
>>> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
>>> writes - but that could have been fixed easily by changing order of
>>> commands too).
>>> >
>>> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com
>>> <mailto:extremecarver at gmail.com>> wrote:
>>> > because you need to do it if you want to show contourlines. Now
>>> compiling the worldwide contourlines each time again - would burn SSD as
>>> well - besides taking longer than just compiling the maps. So everyone who
>>> offers maps for many countries as download puts the contourlines separate.
>>> If you want to offer the choice of showing contourlines or not - mkmgap
>>> will write the gmap contourlines once not needed, and once the maps not
>>> needed. If you don't want to offer that option - it's only the contourlines
>>> being written without need. Contourlines for all geofabrik extracts 20m
>>> equidistance are about 130GB, 10m would be 260GB.
>>> > If you offer 10m and 20m it will be even more not needed writes.
>>> >
>>> > The only solution is to go for a Ramdisk instead - However you likely
>>> will need 128GB of RAM to do that - because for Asia or Europe you need a
>>> 64GB Ramdisk. Same for North America extract (but few people offer that and
>>> just have Canada and the US divided into the 4-5 zones). For other maps you
>>> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
>>> but the windows installers because I don't want to let the server wait for
>>> ages for the single thread NSIS wrapping up the data and instead start the
>>> next country in parallel). And yes going RAMDISK already using my patch and
>>> symlinking those files so mkgmap cannot overwrite them (would still be
>>> faster if mkgmap didn't try in first place I think). If you want to write
>>> out the contourlines as well besides the additional time - for Asia
>>> continent you may then go for a 90GB Ramdisk minimum if offering windows
>>> and gmap installers. In this case gmapsupp.img donwnloads aren't possible
>>> anyhow due to the huge data amounts. For sm
>>> >   aller countries I have them too....
>>> >
>>> > I coded around this now by using symlinks - but yeah that will be
>>> quite a lot of work and is prone to break - while I guess it's only 10
>>> lines or so to add to mkgmap code to have a mode that does not write them
>>> out if they are present - or if you tell on commandline they are present
>>> already. For .img files they aren't overwritten either...
>>> >
>>> > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <
>>> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
>>> wrote:
>>> > Hi Felix,
>>> >
>>> > did not read all the posts in detail. I understand that mkgmap is
>>> neither burning SSDs nor doing any excessive writing unless you call it
>>> multiple times for the same input files. So the question is: Why do you do
>>> that?
>>> >
>>> > Gerd
>>> >
>>> > ________________________________________
>>> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
>>> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
>>> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
>>> > Gesendet: Freitag, 10. September 2021 00:02
>>> > An: Development list for mkgmap
>>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and
>>> converting to disk if used with --gmapi option
>>> >
>>> > Well I think the .tmp files are just building up - and the renamed. So
>>> they are not causing any actual excessive write.
>>> > As for the gmap - it would be cool if there is a mode to not write
>>> them.
>>> >
>>> > Actually it would be great if mkgmap could write all in one go.
>>> Because the thing that takes so much time - is the address search - and
>>> that is always the same. The differences are tiny (just because MapInstall
>>> is crashing when files are missing) you need to compile them separately.
>>> >
>>> > but maybe there could be a mode where mkgamp writes all in one go.
>>> > So family-name / family-name1 / family-name2
>>> > description / description1 / description2
>>> > input input1 input2
>>> > family-name..
>>> > show-profiles
>>> > overview-mapname
>>> > product-id
>>> > (and maybe I missed some options are those that would need to be given
>>> for each set of input tiles. And then just an option where you tell mkgmap
>>> files starting with which first 4 numbers are relevant for address search.
>>> No need to analyze if those other supplied .img (e.g. buildings or
>>> contourlines) need to be added to address search.). I know coded around the
>>> problem of the gmap files causing excessive writes. But yeah that is
>>> actually really complicated be it on windows or linux...
>>> >
>>> > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver at gmail.com
>>> <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:
>>> extremecarver at gmail.com>>> wrote:
>>> > Well I could give it 20 GB ram disk, maybe 32 but then I need to
>>> render on less than 12 processes 64GB ram available). But that is not
>>> enough for Asia continent map, and I guess super tight for Europe...
>>> >
>>> > Mkgmap could definitely keep those .tmp files in memory. But the
>>> important bit is the gmap files not needeed to be written.... Also would
>>> save quite some CPU time.
>>> >
>>> > On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org<mailto:
>>> witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto:
>>> witwall3 at disroot.org>>> wrote:
>>> > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
>>> >> Yes on an nvme disk you barely notice the conversion - it's really
>>> quick.
>>> >> BUT it is not needed if you have the files and even more - it burns
>>> your
>>> >> NVME SSD disk.
>>> > +1. I always use an old spinning rust disk when using mkgmap to save
>>> ssd
>>> > write cycles, even without contours and such. It seems to be profligate
>>> > in its use of disk cycles. I did try using RAM disk, but even with 16GB
>>> > on a laptop, that was soon exhausted.
>>> >
>>> > ael
>>> >
>>> >
>>> > _______________________________________________
>>> > mkgmap-dev mailing list
>>> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
>>> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
>>> mkgmap-dev at lists.mkgmap.org.uk>>
>>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> >
>>> >
>>> > --
>>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>>> >
>>> > _______________________________________________
>>> > mkgmap-dev mailing list
>>> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
>>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> >
>>> >
>>> > --
>>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>>> >
>>> >
>>> >
>>> > --
>>> > Felix Hartman - Openmtbmap.org & VeloMap.org
>>> >
>>> > _______________________________________________
>>> > mkgmap-dev mailing list
>>> > mkgmap-dev at lists.mkgmap.org.uk
>>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>
>>>
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>>
>>
>> --
>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>
>>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210913/a39a3870/attachment-0001.html>