[mkgmap-dev] Regression causing Heap space exhaustion?

Sat Aug 22 11:51:07 BST 2009

Morten Kjeldgaard <mok at bioxray.au.dk> writes:

> On 22/08/2009, at 01.53, Greg Troxel wrote:
>
>> With splitter/mkgmap, the general strategy I use is:
>>
>>  set heap size to a bit less than RAM, to avoid paging
>>
>>  use --max-nodes to splitter to make small enough areas.
>>
>>
>> Here is the script I use (with a mac that has 4G physical ram):
>
> Great! Thanks, I will explore this and report back!
>
> In the meantime, I've run mkgmap on some of the files from geofabrik,  
> here are the results when running the current svn version, using:
>
>   java -Xmx1024m -jar mkgmap file.osm
>
> Size		File						Result
> ----------------------------------------------------------------------
> 2,2G		denmark.osm			Fail

This is a confused thing to try.  mkgmap can only process so much data
in one tile.  Use the splitter.

> 474M		63240001.osm dk, split1	Fail
> 654M		norway.osm				Fail
> 629M		poland.osm				Fail
> 289M		belgium.osm				OK
> 266M		romania.osm				OK
> 59M		luxembourg.osm			OK
> 4,8M		andorra.osm				OK
> -----------------------------------------------------------------------
>
> Although it seems that the problem is somewhat related to file size,  
> there is no simple interpretation possible, since file #2 is less than  
> twice the size of belgium.osm, but that fails even with 4x the amount  
> of RAM for the VM. Also, there are probably great differences in the  
> use of various tags in the different areas of the maps. Perhaps  
> someone can see another pattern here.

Input files contain nodes, ways, relations, and some of them cause
objects in the output and some don't.  So there are no neat formulas.
But definitely more data leads to more memory and if you don't have
enough it doesn't work.  This is not mysterious and is well known in the
mkgmap world.  Documenting approximate sizes is quite useful, but
worrying exactly about the edges is in my opinion not useful.  The
normal way is to use the splitter, and so the key question is what
splitter parameters to give.

So,

  set the heap size to a bit less than physram and leave it there
  (e.g. 1700k for 2M physical) to avoid paging

  use the splitter

    max-nodes 1600000 might be a good value, and if you have trouble
    report it.  I have had people say that doesn't work with --route and
    they need to use 800000, but 1.6M works for me.

  if you are inclined to explore this further, read the source and add
  accounting of resources and printouts at end of tile to mkgmap and
  send patches.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20090822/59006b82/attachment.bin 