<html><head></head><body><div dir="auto">Hi Gerd,<br></div>
<div dir="auto">Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap per job. I will check on the weekend. Can you please post a patched mkgmap.jar? <br><br></div>
<div dir="auto">How about doing it the other way round and suggest the user to use max-job option if mkgmap think it is useful?<br><br></div>
<div dir="auto">Henning </div>
<div class="gmail_quote" >On 8 Feb 2018, at 16:38, Gerd Petermann <<a href="mailto:gpetermann_muenchen@hotmail.com" target="_blank">gpetermann_muenchen@hotmail.com</a>> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="blue">Hi Manfred,<br><br>thanks for testing. Just to make this clear: I always use --max-jobs, because I also<br>always use -Xmx whenever I compile many tiles.<br><br>In this case a large tile is one that results in a large *.img<br>(one with many MB). Typically this depends on the --max-nodes option for splitter<br>and the style and the mkgmap options (routable or not, without or without DEM etc).<br><br>The max-jobs option allows to specify the number of parallel jobs,<br>e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one input file is done,<br>the next one is started until all are finished. The patch only changes the default from 1<br>to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but this will slow down<br>the run time.<br><br>What I tried to point out is that a user who doesn't know about the --max--jobs option<br>might as well not know about the meaning of the -Xmx JRE option.<br><br>What happens if you start mkgmap on an 8-core machine with max-jobs=8 and<br>without -Xmx option? I think it will crash, and with the 2nd version of the patch it will tell<br>the user to look at the -Xmx option. So far so good.<br>What happens on a 4 core machine with max-jobs=4 and no -Xmx option?<br>What if the OS is 32 bit only? This would not allow much more than -Xmx1700m.<br>What if the option is used but set to a rather low value, e.g. -Xmx=1000m?<br><br>My current thinking is this:<br>If --max-jobs is not specified mkgmap should check the number of cores and the available heap<br>and assume that one thread needs ~500M. <br><br>So, I would use something like num = Math.min(Runtime.getRuntime().maxMemory() / (512 * 1024 * 1024),<br> Runtime.getRuntime().availableProcessors()) as default, maybe also for the case that one<br>specifies --max-jobs .<br><br>How does that sound?<br><br>Gerd<br><br><br><br><br><br><br><hr><br>Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Manfred Haiduk <mhaiduk@t-online.de><br>Gesendet: Mittwoch, 7. Februar 2018 22:28<br>An: mkgmap-dev@lists.mkgmap.org.uk<br>Betreff: Re: [mkgmap-dev] max-jobs patch<br><br>I don't know, if Nordrhein-Westfalen is that kind of large tiles you<br>meant. Splitter ended up with 43 files and i tried to make mapsource<br>maps with and without --max-jobs option and mkgmap r4000. My computer<br>does the job with 43 files about 12minutes faster than without this<br>option. My laptop is a core i5 with 8Gb of memory. The command i use is<br>like this line :<br><br>java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar<br>-Dlog.config=<a href="http://logging.properties">logging.properties</a><br>C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar<br>--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006<br>.\styles\mystyle.txt --series-name="STYLE TEST"<br>--family-name="OpenStreetmap" --country-name=STYLETEST<br>--country-abbr=STY --area-name=STY --overview-mapname="overview"<br>--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going<br>--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE<br>--index --bounds=mybounds --route<br>--name-tag-list=name,place_name,loc_name --housenumbers<br>--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip<br>--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args<br>--description=MYSTYLE --remove-ovm-work-files=true<br><br>Hope this gave you some useful hints<br><br><br>Am 07.02.2018 um 20:06 schrieb Gerd Petermann:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> Hi Mike,<br><br> I did not yet try it. My understanding is that the Garbage Collection will try hard<br> to avoid an out of heap exception. In other words: A user might see better throughput<br> with fewer parallel jobs. What are your experiences? Did you try with<br> large maps, say > 40 large tiles ?<br><br> Gerd<br><br><hr><br> Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk><br> Gesendet: Mittwoch, 7. Februar 2018 01:58<br> An: 'Development list for mkgmap'<br> Betreff: Re: [mkgmap-dev] max-jobs patch<br><br> Hi Gerd, please find attached v2 of the max-jobs patch.<br><br> This version additionally catches out of heap memory exceptions and displays<br> a message suggesting either the use of the Java -Xmx option to increase the<br> available heap memory or the mkgmap --max-jobs option with a smaller value<br> to reduce the memory requirement (the latter is suggested only if using more<br> than one thread).<br><br> How does that seem?<br> Regards,<br> Mike<br><br> -----Original Message-----<br> From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com]<br> Sent: 05 February 2018 07:28<br> To: Mike Baggaley <mike@tvage.co.uk>; 'Development list for mkgmap'<br> <mkgmap-dev@lists.mkgmap.org.uk><br> Subject: Re: [mkgmap-dev] max-jobs patch<br><br> Hi Mike,<br><br> thought about this again. Maybe this change is too simple. With multiple<br> jobs one also<br> needs more heap (-Xmx JRE option). If you create rather large tiles with<br> splitter (max-nodes=1600000)<br> you need 0.5 - 1 GB for each job. Not sure what happens when a user creates<br> a map with<br> 10 tiles on an 8-core machine without any -Xmx option.<br><br> I fear this will result in OutOfMemory exception, so better check the<br> available heap as well.<br><br> Gerd<br><br><hr><br> Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike<br> Baggaley <mike@tvage.co.uk><br> Gesendet: Sonntag, 4. Februar 2018 15:14<br> An: 'Development list for mkgmap'<br> Betreff: [mkgmap-dev] max-jobs patch<br><br> Hi Gerd,<br><br> Please find attached a patch that amends the default behaviour if the<br> --max-jobs option is not specified, to using a value equal to the number of<br> CPU cores, as suggested in a previous post. The documentation is also<br> amended to reflect the change. This halves the execution time of mkgmap for<br> building a map of Staffordshire on my 8-core PC when --max-jobs is not<br> specified (I didn't know about this option previously and was unaware the<br> performance could have been improved).<br><br> Cheers,<br> Mike<br><br><hr><br> mkgmap-dev mailing list<br> mkgmap-dev@lists.mkgmap.org.uk<br> <a href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a></blockquote><br>--<br><br> Mit freundlichen Grüßen<br><br>#####################################################<br>Manfred Haiduk,<br>Zum Fischbach 9,<br>52393 Hürtgenwald<br>e-mail mhaiduk@t-online.de<br>#####################################################<br><br><hr><br>mkgmap-dev mailing list<br>mkgmap-dev@lists.mkgmap.org.uk<br><a href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a><br><hr><br>mkgmap-dev mailing list<br>mkgmap-dev@lists.mkgmap.org.uk<br><a href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a><br><br></pre></blockquote></div></body></html>