[mkgmap-dev] max-jobs patch

Sat Feb 10 09:53:05 GMT 2018

Hi Gerd, I have put together the attached patch that should result in a
reasonable value for max-jobs if it is defaulted. This version attempts to
calculate a suitable value by processing one task, seeing how much memory it
used, then dividing the available memory by the memory used to determine the
value before running the remaining tasks. I have tried a variety of values
of -Xmx on my PC, and it seems to work for both a county and the whole of
the UK. I haven't tried it on a continent, as I don't have suitable data. I
can't guarantee that every combination of memory and CPU will work, but if
it does fail, you will still get a sensible error message telling you what
to change. At the end of the run, if insufficient memory is allocated to the
Java heap to allow all the CPU cores to be used, it will suggest a value for
-Xmx. For my maps, it seems to suggest about 400Mb per thread. My build of
the UK without specifying --max-jobs has reduced from about 3 hours to 45
minutes.

I have also made a few further minor improvements to the documentation to
make the form more consistent, and changed the logged execution time from
milliseconds to a more useful hours, minutes and seconds (or milliseconds
when less than 1 second).

Regards,
Mike

-----Original Message-----
From: Gerd Petermann [mailto:gpetermann_muenchen at hotmail.com] 
Sent: 07 February 2018 19:06
To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will
try hard
to avoid an out of heap exception. In other words: A user might see better
throughput
with fewer parallel jobs. What are your experiences? Did you try with
large maps, say > 40 large tiles ?

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <mike at tvage.co.uk>
Gesendet: Mittwoch, 7. Februar 2018 01:58
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-----Original Message-----
From: Gerd Petermann [mailto:gpetermann_muenchen at hotmail.com]
Sent: 05 February 2018 07:28
To: Mike Baggaley <mike at tvage.co.uk>; 'Development list for mkgmap'
<mkgmap-dev at lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=1600000)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <mike at tvage.co.uk>
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike

-------------- next part --------------
A non-text attachment was scrubbed...
Name: maxjobs-v3.patch
Type: application/octet-stream
Size: 24330 bytes
Desc: not available
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20180210/eaceb4d9/attachment-0001.obj>