[mkgmap-dev] Performance with large files

Tue Mar 21 17:56:51 GMT 2017

Hi Gerd,

mkgmap is running now, I'll likely report tomorrow.

But I already saw that the first tile took very long, and in the areas 
file it is:
43100001: 1765376,-935936 to 2822144,139264
#       : 37.880859,-20.083008 to 60.556641,2.988281

So, I suspect that here could be a problem: some of the newly added 
extracts inflates the area covered by the map extremely. I first thought 
of the Dutch Antilles in the Caribean, but that's farther west and 
farther south. I cannot see how that area comes into the map.

The maps I created before, typically contained Germany, Chechia, 
Austria, Liechtenstein, and Switzerland. That took normally less than an 
hour per map.

Now I added Luxemburg, Belgium and Netherlands also, which does not 
increase file size a lot. But the time for map creation was increased 
enormously.

Bernhard

Am 21.03.2017 um 09:17 schrieb Gerd Petermann:
> Hi Bernhard,
>
> other tests did not show new results. Any idea why you got so different numbers?
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen at hotmail.com>
> Gesendet: Montag, 20. März 2017 07:40:00
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Performance with large files
>
> Hi Bernhard,
>
> reg. splitter:
> If I got that right you used pbf as input format for the "Germany" result and *.o5m for "Central Europe".
> Since the o5m format allows faster processing the splitter times are okay for me.
>
> reg. mkgmap:
> I've added a few lines in mkgmap to report the calculation time for each tile. As you might know,
> mkgmap starts a few threads (depending on the max-jobs option) and each thread processes on tile at a time.
> The threads share only a few data structures, and mkgmap doesn't collect much information for each tile in
> memory, so  there is no good reason for an increase of run time per processed tile.
>
> My system is probably close to yours, a machine with 8 GB Memory and a 4 core CPU (i5) running a 64 Windows.
>
> The attached patch adds a few lines to report the calculation time for a tile. The patched version is here:
> http://files.mkgmap.org.uk/download/339/mkgmap.jar
>
>   I've used the patched version to compile a map with the OpenfietsMap Lite style of an area around (and including) Germany .
> I used the attached dach-x.poly file with osmconvert and a planet.o5m file from 2017-01-17.
> and splitter with max-nodes00000 & keep-complete=true and found what I expected.
> Times are between 7 and 35 seconds per tile, most are around ~20 secs, no increase in time/tile.
> The time for the creation of the index / gmapsupp is rather short compared to the overall run time.
> The complete mkgmap log is here :
> http://files.mkgmap.org.uk/download/340/mkgmap_2017-03-20-064126.log
>
> So, I cannot reproduce your result for mkgmap.
> Maybe your machine was busy with other work like installing updates, maybe some tiles in the added area
> require much more time, maybe times depend on program options or style.
>
> I just noticed that I did not enable assertions (-ea) so  I now try a variant with a max-nodes00000 and -ea.
>
> Gerd
>
>
>
>
>
>
>
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil at gmx.de>
> Gesendet: Sonntag, 19. März 2017 19:46:32
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Performance with large files
>
> Eventually I updated Java and tried the latest version of mkgmap 3847
> (and also splitter 580).
> The extracts were retrieved from Geofabrik on Saturday 18.
> Germany: 2.93 GB
> plus additional countries: 5.62 GB (incl. Germany) which were combined
> to a 7.57 GB o5m file.
> splitter: Germany 14 minutes - Central Europe 20 minutes
> mkgmap: Germany 30 minutes - Central Europe 133 minutes
> While splitter performed better than O(n) (more like O(sqrt(n))),
> mkgmap performed worse than O(n^2).
>
> Am 19.03.2017 um 12:06 schrieb Bernhard Hiller:
>> How is mkgmap expected to behave when input files grow in size? Is a
>> linear inrease in calculation time - i.e. O(n) - expected, or an
>> increase beyond linearity?
>> E.g. when I create a map with routable lines for bicycle, mkgmap takes
>> some 30 minutes for Germany alone (3 GB pbf file resulting in 850 MB
>> img file), but more than  2 hours for Germany and some neighboring
>> countries (7 GB o5m file, resulting 1.4 GB img).
>> Are there many calculations at O(n^2) or beyond in mkgmap, or is this
>> due to other factors, e.g. memory limitation?
>> Notes:
>> mkgmap is called by
>> %JAVA% -Xmx6800M -ea -jar %MKGMAP% ....
>> on 64bit Win 7;
>> swapping to disc does not occur.
>> But I am more interested in a general rule than in some hints for
>> improving the performance in this concrete case. E.g. how I could
>> estimate the duration if I add some further countries...
>> Thanks for your hints.
>>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>