[mkgmap-dev] Memory Consumption on Index Creation

Mon Aug 29 21:07:42 BST 2011


On 29.08.2011 20:52, WanMil wrote:
>> 1. I noticed that (at least for my style-file) 7500MB of RAM available
>> does not suffice to compile a map of Europe including index! However if
>> I recreate the index in another go, then it is no problem.
>>
>> How comes? The index creation seems to be completly independent of the
>> maptiles creation. Is mkgmap leaving stuff inside the memory, that is in
>> reality not needed anymore?
>
> I think you have listed your style and your mkgmap parameters on your 
> website. Can you give a link to them? That makes it easier to find the 
> problem.


    Subversion

  * Path: https://svn.origo.ethz.ch/openmtbmap/
  * Web SVN: http://svn.origo.ethz.ch/wsvn/openmtbmap/
  * ViewVC: http://svn.origo.ethz.ch/viewvc/openmtbmap/
  * Nightly SVN Dump: http://svn.origo.ethz.ch/dump/openmtbmap/latest.tar.gz


And here is the specific options on full creation (crashes low on java 
heap space, passes if Xmx11000M is given, but I only have 12GB RAM on a 
testmachine on my build sever there is only about 7800M before it 
seriously slows down -- this is running against Europe extract from 
geofabrik):
c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M 
c:\openmtbmap\mkgmap.jar --max-jobs=4 
--generate-sea=extend-sea-sectors,close-gaps=6000,floodblocker,fbgap=60,fbthres=200,fbratio=0.6 
--latin1 "--style-file=c:\openmtbmap\new4"  --reduce-point-density=4 
--nsis --index --transparent --adjust-turn-headings --add-pois-to-areas 
--ignore-maxspeeds --x-reduce-point-density-polygon=8 
--link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 
--remove-short-arcs=4 --description=openmtbmap_eu --merge-l
ines --location-autofill=1 --route --country-abbr=eu 
--country-name=europe  --mapname=65500000 --family-id=6550 
--product-id=1 --series-name=openmtbmap_europe_29.08.2011 
--family-name=mtbmap_eu_29.08.2011 --tdbfile --overview-mapname=mapset 
--keep-going --area-name="europe_29.08.2011_openmtbmap.org"  -c 
c:\openmtbmap\maps\template.europe  1>NUL

versus the options for index only creation (works well, I can even 
reduce Xmx to about 4500M):
c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M 
c:\openmtbmap\mkgmap.jar --nsis --index --transparent 
--description=openmtbmap_eu --location-autofill=1 --country-abbr=eu 
--country-name=europe  --mapname=65500000 --family-id=6550 
--product-id=1 --series-name=openmtbmap_europe_29.08.2011 
--family-name=mtbmap_eu_29.08.2011
--tdbfile --overview-mapname=mapset --keep-going 
--area-name="europe_29.08.2011_openmtbmap.org" 6550*.img
>
> And important: Do you use any patches?
yes listed here:
http://svn.origo.ethz.ch/wsvn/openmtbmap/mkgmap_patches.patch

but they shouldn't affect the index creation. Maybe it's again an issue 
of assigning names like ft to unnamed footways. However there must still 
be some memory leak if creating the index standalone it needs only about 
50% of memory. Actually it would be really great if the index creation 
could be lighter on memory, maybe using a temporary file for buffer?
>
> WanMil
>
>
>>
>> 2. Also as running out of memory primarily happens on index creation, it
>> would be great if mkgmap writes out the index under a temporary name,
>> and once it has passed correctly, renames the file. Else it is not
>> possible to easily assess in a bash/batch script whether mkgmap
>> completed the map correctly or not (and hence one uploads broken maps if
>> everything automated). Or at least I don't see any way to find out
>> whether it passed correctly or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20110829/e640f000/attachment.html