[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
From Carlos Dávila carlos at alternativaslibres.org on Sun Sep 12 20:34:05 BST 2021
I share the same issue exposed by Felix, but not only for contour lines but also for DEM, which I have precompiled as *.img for many countries and just combine with the updated OSM tiles. El 12/9/21 a las 19:10, Felix Hartmann escribió: > And well - it is burning SSD for the contourlines currently even if > not calling multiple times. 130GB minimum per worldwide map update for > all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling > multiple times and 10m and 20m I had about 2TB of useless writes per > weekly map update. I've got rid of all of them with my uncommenting > the line, plus saved about 500GB of writes by now doing all the > gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk > writes per map update I'm down to 1TB.. (and yeah the resplitting of > tiles added another 5TB of writes - but that could have been fixed > easily by changing order of commands too). > > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com>> wrote: > > because you need to do it if you want to show contourlines. Now > compiling the worldwide contourlines each time again - would burn > SSD as well - besides taking longer than just compiling the maps. > So everyone who offers maps for many countries as download puts > the contourlines separate. If you want to offer the choice of > showing contourlines or not - mkmgap will write the gmap > contourlines once not needed, and once the maps not needed. If you > don't want to offer that option - it's only the contourlines being > written without need. Contourlines for all geofabrik extracts 20m > equidistance are about 130GB, 10m would be 260GB. > If you offer 10m and 20m it will be even more not needed writes. > > The only solution is to go for a Ramdisk instead - However you > likely will need 128GB of RAM to do that - because for Asia or > Europe you need a 64GB Ramdisk. Same for North America extract > (but few people offer that and just have Canada and the US divided > into the 4-5 zones). For other maps you will get by with a 15-20GB > Ramdisk (which I have resorted to now for all but the windows > installers because I don't want to let the server wait for ages > for the single thread NSIS wrapping up the data and instead start > the next country in parallel). And yes going RAMDISK already using > my patch and symlinking those files so mkgmap cannot overwrite > them (would still be faster if mkgmap didn't try in first place I > think). If you want to write out the contourlines as well besides > the additional time - for Asia continent you may then go for a > 90GB Ramdisk minimum if offering windows and gmap installers. In > this case gmapsupp.img donwnloads aren't possible anyhow due to > the huge data amounts. For smaller countries I have them too.... > > I coded around this now by using symlinks - but yeah that will be > quite a lot of work and is prone to break - while I guess it's > only 10 lines or so to add to mkgmap code to have a mode that does > not write them out if they are present - or if you tell on > commandline they are present already. For .img files they aren't > overwritten either... > > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann > <gpetermann_muenchen at hotmail.com > <mailto:gpetermann_muenchen at hotmail.com>> wrote: > > Hi Felix, > > did not read all the posts in detail. I understand that mkgmap > is neither burning SSDs nor doing any excessive writing unless > you call it multiple times for the same input files. So the > question is: Why do you do that? > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk > <mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag > von Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com>> > Gesendet: Freitag, 10. September 2021 00:02 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and > converting to disk if used with --gmapi option > > Well I think the .tmp files are just building up - and the > renamed. So they are not causing any actual excessive write. > As for the gmap - it would be cool if there is a mode to not > write them. > > Actually it would be great if mkgmap could write all in one > go. Because the thing that takes so much time - is the address > search - and that is always the same. The differences are tiny > (just because MapInstall is crashing when files are missing) > you need to compile them separately. > > but maybe there could be a mode where mkgamp writes all in one go. > So family-name / family-name1 / family-name2 > description / description1 / description2 > input input1 input2 > family-name.. > show-profiles > overview-mapname > product-id > (and maybe I missed some options are those that would need to > be given for each set of input tiles. And then just an option > where you tell mkgmap files starting with which first 4 > numbers are relevant for address search. No need to analyze if > those other supplied .img (e.g. buildings or contourlines) > need to be added to address search.). I know coded around the > problem of the gmap files causing excessive writes. But yeah > that is actually really complicated be it on windows or linux... > > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann > <extremecarver at gmail.com > <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com > <mailto:extremecarver at gmail.com>>> wrote: > Well I could give it 20 GB ram disk, maybe 32 but then I need > to render on less than 12 processes 64GB ram available). But > that is not enough for Asia continent map, and I guess super > tight for Europe... > > Mkgmap could definitely keep those .tmp files in memory. But > the important bit is the gmap files not needeed to be > written.... Also would save quite some CPU time. > > On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org > <mailto:witwall3 at disroot.org><mailto:witwall3 at disroot.org > <mailto:witwall3 at disroot.org>>> wrote: > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote: > > > > Yes on an nvme disk you barely notice the conversion - it's > really quick. > > BUT it is not needed if you have the files and even more - > it burns your > > NVME SSD disk. > > +1. I always use an old spinning rust disk when using mkgmap > to save ssd > write cycles, even without contours and such. It seems to be > profligate > in its use of disk cycles. I did try using RAM disk, but even > with 16GB > on a laptop, that was soon exhausted. > > ael > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk>> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
- Previous message: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
- Next message: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list