[mkgmap-dev] MDR building out-of-memory

Mon May 10 22:05:49 BST 2021

Hi Gerd

Since downloading loading britain-and-ireland-latest.osm.pbf I had been
unable to build a gmapsupp because of running out of heap (my hardware
is 32 bit, -Xmx1540M is largest value allowed)

My problem is mainly because I have 1731146 cities (along with 1046096
streets)

Looking at Mdr5 processing, I've changed it in 3 ways to improve memory
usage and garbage collection.

1/ use trimToSize() after all the cities are loaded from the individual
tile .img. I presume that the growth factor gradually increases as it
runs out of allocated array space. I had to change the declaration from
List<Mdr5Record> to ArrayList<Mdr5Record> to allow this, but I can't
see any problem in this.

2/ Move the main part of preWriteImpl into its own method so the first
sortKeys ArrayList and Sort can be freed before calcMdr20/1/2() each
create another massive SortKeys and Sort.

3/ Move the scope of mdr20s to a class variable. This is referenced by
all the Mdr5Records and the scope of where it was declared before
seemed to to cause the garbage collector major problems - it churned
for 5 mins using all the processors before running out of memory. After
moving it, the whole of mdr is built in a couple of mins with cpu usage
mostly < 125%.

Ticker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mdr5Mem.patch
Type: text/x-patch
Size: 2212 bytes
Desc: not available
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210510/833f59c9/attachment.bin>