[mkgmap-dev] max-jobs patch

Thu Feb 8 14:34:23 GMT 2018

Hi Henning,

maybe you see those high heap values with DEM? How do measure it?
I think the monitor function of VisualVM shows quite well if Garbage Collection is under stress or not.

binary is here:
http://files.mkgmap.org.uk/download/415/mkgmap.jar

For those users who already used --max-jobs with or without a paramter before there should be no difference.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm at hscholland.de>
Gesendet: Donnerstag, 8. Februar 2018 09:56
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd,
Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap per job. I will check on the weekend. Can you please post a patched mkgmap.jar?

How about doing it the other way round and suggest the user to use max-job option if mkgmap think it is useful?

Henning
On 8 Feb 2018, at 16:38, Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> wrote:

Hi Manfred,

thanks for testing. Just to make this clear: I always use --max-jobs, because I also
always use -Xmx whenever I compile many tiles.

In this case a large tile is one that results in a large *.img
(one with many MB). Typically this depends on the --max-nodes option for splitter
and the style and the mkgmap options (routable or not, without or without DEM etc).

The max-jobs option allows to specify the number of parallel jobs,
e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one input file is done,
the next one is started until all are finished. The patch only changes the default from 1
to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but this will slow down
the run time.

What I tried to point out is that a user who doesn't know about the --max--jobs option
might as well not know about the meaning of the -Xmx JRE option.

What happens if you start mkgmap on an 8-core machine with max-jobs=8 and
without -Xmx option? I think it will crash, and with the 2nd version of the patch it will tell
the user to look at the -Xmx option. So far so good.
What happens on a 4 core machine with max-jobs=4 and no -Xmx option?
What if the OS is 32 bit only? This would not allow much more than -Xmx1700m.
What if the option is used but set to a rather low value, e.g. -Xmx=1000m?

My current thinking is this:
If --max-jobs is not specified mkgmap should check the number of cores and the available heap
and assume that one thread needs ~500M.

So, I would use something like num = Math.min(Runtime.getRuntime().maxMemory() / (512 * 1024 * 1024),
 Runtime.getRuntime().availableProcessors()) as default, maybe also for the case that one
specifies --max-jobs .

How does that sound?

Gerd

________________________________

Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Manfred Haiduk <mhaiduk at t-online.de>
Gesendet: Mittwoch, 7. Februar 2018 22:28
An: mkgmap-dev at lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I don't know, if Nordrhein-Westfalen is that kind of large tiles you
meant. Splitter ended up with 43 files and i tried to make mapsource
maps with and without --max-jobs option and mkgmap r4000. My computer
does the job with 43 files about 12minutes faster than without this
option. My laptop is a core i5 with 8Gb of memory. The command i use is
like this line :

java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar
-Dlog.config=logging.properties<http://logging.properties>
C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar
--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006
.\styles\mystyle.txt --series-name="STYLE TEST"
--family-name="OpenStreetmap" --country-name=STYLETEST
--country-abbr=STY --area-name=STY --overview-mapname="overview"
--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going
--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE
--index --bounds=mybounds --route
--name-tag-list=name,place_name,loc_name --housenumbers
--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip
--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args
--description=MYSTYLE --remove-ovm-work-files=true

Hope this gave you some useful hints

Am 07.02.2018 um 20:06 schrieb Gerd Petermann:
 Hi Mike,

 I did not yet try it. My understanding is that the Garbage Collection will try hard
 to avoid an out of heap exception. In other words: A user might see better throughput
 with fewer parallel jobs. What are your experiences? Did you try with
 large maps, say > 40 large tiles ?

 Gerd

________________________________

 Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike at tvage.co.uk>
 Gesendet: Mittwoch, 7. Februar 2018 01:58
 An: 'Development list for mkgmap'
 Betreff: Re: [mkgmap-dev] max-jobs patch

 Hi Gerd, please find attached v2 of the max-jobs patch.

 This version additionally catches out of heap memory exceptions and displays
 a message suggesting either the use of the Java -Xmx option to increase the
 available heap memory or the mkgmap --max-jobs option with a smaller value
 to reduce the memory requirement (the latter is suggested only if using more
 than one thread).

 How does that seem?
 Regards,
 Mike

 -----Original Message-----
 From: Gerd Petermann [mailto:gpetermann_muenchen at hotmail.com]
 Sent: 05 February 2018 07:28
 To: Mike Baggaley <mike at tvage.co.uk>; 'Development list for mkgmap'
 <mkgmap-dev at lists.mkgmap.org.uk>
 Subject: Re: [mkgmap-dev] max-jobs patch

 Hi Mike,

 thought about this again. Maybe this change is too simple. With multiple
 jobs one also
 needs more heap (-Xmx JRE option). If you create rather large tiles with
 splitter (max-nodes=1600000)
 you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
 a map with
 10 tiles on an 8-core machine without any -Xmx option.

 I fear this will result in OutOfMemory exception, so better check the
 available heap as well.

 Gerd

________________________________

 Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Mike
 Baggaley <mike at tvage.co.uk>
 Gesendet: Sonntag, 4. Februar 2018 15:14
 An: 'Development list for mkgmap'
 Betreff: [mkgmap-dev] max-jobs patch

 Hi Gerd,

 Please find attached a patch that amends the default behaviour if the
 --max-jobs option is not specified, to using a value equal to the number of
 CPU cores, as suggested in a previous post. The documentation is also
 amended to reflect the change. This halves the execution time of mkgmap for
 building a map of Staffordshire on my 8-core PC when --max-jobs is not
 specified (I didn't know about this option previously and was unaware the
 performance could have been improved).

 Cheers,
 Mike

________________________________

 mkgmap-dev mailing list
 mkgmap-dev at lists.mkgmap.org.uk
 http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--

  Mit freundlichen Grüßen

#####################################################
Manfred Haiduk,
Zum Fischbach 9,
52393 Hürtgenwald
e-mail mhaiduk at t-online.de
#####################################################

________________________________

mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
________________________________

mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev