[mkgmap-dev] [PATCH v5]splitter memory usage

Thu Nov 17 20:52:01 GMT 2011

Sorry, all of my my previous measurements were actually done with 
splitter r188. So also the fastutil.jar missing problem was of course 
caused by it.
I have updated all tests with r189. Note that optimize-mem at least on 
my system (Core7 with 4 Cores available, 8GB Ram, Win 7 x64 server) is 
only faster on Europe.pbf. Else it is slower. r189 is in general faster 
or as fast as r188. However using optimize-mem on Germany.pbf it slowed 
down a bit (was faster without optimize-mem though). As most people will 
not split files as big as europe.pbf very often, I think 
optimize-mem=false should stay the default.

Therefore I reran some tests

1 to complement old test:
splitter r188 without optimize-mem and max-nodes 1 100 000 instead of 
900 000 (807 areas instead of 975)): 1678 seconds (so 107 seconds gained 
over smaller max-nodes).
splitter r189 without optimize-mem (max-nodes 1 100 000):  1635 seconds
splitter r189 with optimize-mem (max-nodes 1 100 000): 1588 seconds

Germany
splitter r189 with optimize-mem: 190 seconds
splitter r189 without optimize-mem 173
splitter r188 with optimize-mem: 186
splitter r188 without optimize-mem: 182
(just as comparison noted earlier: splitter 181:  225seconds)

Austria
splitter r189 with optimize-mem: 36
splitter r189 without optimize-mem 34
splitter r188 with optimize-mem: 34
splitter r188 without optimize-mem: 33
r181: 43 seconds

Some Statistics on Europe with optimize-mem (without below)
Final statistics
***********************************************************
coords occupancy
MAP occupancy: 579'162'289, number of area dictionary entries: 5596
Length-8 chunks: 5'243'245 (Bytes: 146'810'860)
Length-12 chunks: 1'556'563 (Bytes: 56'036'268)
Length-16 chunks: 873'095 (Bytes: 38'416'180)
Length-20 chunks: 677'690 (Bytes: 35'239'880)
Length-24 chunks: 604'329 (Bytes: 36'259'740)
Length-28 chunks: 568'332 (Bytes: 38'646'576)
Length-32 chunks: 564'759 (Bytes: 42'921'684)
Length-36 chunks: 580'701 (Bytes: 48'778'884)
Length-40 chunks: 580'432 (Bytes: 53'399'744)
Length-44 chunks: 588'583 (Bytes: 58'858'300)
Length-48 chunks: 605'760 (Bytes: 65'422'080)
Length-52 chunks: 567'664 (Bytes: 65'849'024)
Length-56 chunks: 493'862 (Bytes: 61'238'888)
Length-60 chunks: 434'713 (Bytes: 57'382'116)
Length-64 chunks: 465'204 (Bytes: 65'128'560)
Length-68 chunks: 1'481'852 (Bytes: 219'314'096)
RLE compresion info: compressed / uncompressed size / ratio: 449'530'736 
/ 664'184'788 / 33%
Sparse chunk vector details: used 15'886'784 of 20'631'616 allocated 
entries (< 78%)
ways occupancy
MAP occupancy: 68'744'281, number of area dictionary entries: 8438
Length-8 chunks: 401'162 (Bytes: 11'232'536)
Length-12 chunks: 188'797 (Bytes: 6'796'692)
Length-16 chunks: 124'579 (Bytes: 5'481'476)
Length-20 chunks: 103'673 (Bytes: 5'390'996)
Length-24 chunks: 93'180 (Bytes: 5'590'800)
Length-28 chunks: 86'282 (Bytes: 5'867'176)
Length-32 chunks: 81'099 (Bytes: 6'163'524)
Length-36 chunks: 76'133 (Bytes: 6'395'172)
Length-40 chunks: 72'515 (Bytes: 6'671'380)
Length-44 chunks: 69'720 (Bytes: 6'972'000)
Length-48 chunks: 66'263 (Bytes: 7'156'404)
Length-52 chunks: 61'777 (Bytes: 7'166'132)
Length-56 chunks: 55'173 (Bytes: 6'841'452)
Length-60 chunks: 47'582 (Bytes: 6'280'824)
Length-64 chunks: 47'710 (Bytes: 6'679'400)
Length-68 chunks: 165'749 (Bytes: 24'530'852)
RLE compresion info: compressed / uncompressed size / ratio: 52'160'044 
/ 78'078'696 / 34%
Sparse chunk vector details: used 1'741'394 of 2'005'472 allocated 
entries (< 88%)

Thread worker-2 has finished
Thread worker-0 has finished
Thread worker-1 has finished
Time finished: Thu Nov 17 21:09:39 CET 2011
Total time taken: 1588s

Some Statistics on Europe (without optimize-mem):
Final statistics
***********************************************************
coords occupancy
MAP occupancy: 579'162'289, number of area dictionary entries: 5596
Length-8 chunks: 5'243'245 (Bytes: 146'810'860)
Length-12 chunks: 1'556'563 (Bytes: 56'036'268)
Length-16 chunks: 873'095 (Bytes: 38'416'180)
Length-20 chunks: 677'690 (Bytes: 35'239'880)
Length-24 chunks: 604'329 (Bytes: 36'259'740)
Length-28 chunks: 568'332 (Bytes: 38'646'576)
Length-32 chunks: 564'759 (Bytes: 42'921'684)
Length-36 chunks: 580'701 (Bytes: 48'778'884)
Length-40 chunks: 580'432 (Bytes: 53'399'744)
Length-44 chunks: 588'583 (Bytes: 58'858'300)
Length-48 chunks: 605'760 (Bytes: 65'422'080)
Length-52 chunks: 567'664 (Bytes: 65'849'024)
Length-56 chunks: 493'862 (Bytes: 61'238'888)
Length-60 chunks: 434'713 (Bytes: 57'382'116)
Length-64 chunks: 465'204 (Bytes: 65'128'560)
Length-68 chunks: 1'481'852 (Bytes: 219'314'096)
RLE compresion info: compressed / uncompressed size / ratio: 449'530'736 
/ 664'184'788 / 33%
Chunk vector details: used 15'886'784 of 33'554'432 allocated entries (< 
48%)
ways occupancy
MAP occupancy: 68'744'281, number of area dictionary entries: 8438
Length-8 chunks: 401'162 (Bytes: 11'232'536)
Length-12 chunks: 188'797 (Bytes: 6'796'692)
Length-16 chunks: 124'579 (Bytes: 5'481'476)
Length-20 chunks: 103'673 (Bytes: 5'390'996)
Length-24 chunks: 93'180 (Bytes: 5'590'800)
Length-28 chunks: 86'282 (Bytes: 5'867'176)
Length-32 chunks: 81'099 (Bytes: 6'163'524)
Length-36 chunks: 76'133 (Bytes: 6'395'172)
Length-40 chunks: 72'515 (Bytes: 6'671'380)
Length-44 chunks: 69'720 (Bytes: 6'972'000)
Length-48 chunks: 66'263 (Bytes: 7'156'404)
Length-52 chunks: 61'777 (Bytes: 7'166'132)
Length-56 chunks: 55'173 (Bytes: 6'841'452)
Length-60 chunks: 47'582 (Bytes: 6'280'824)
Length-64 chunks: 47'710 (Bytes: 6'679'400)
Length-68 chunks: 165'749 (Bytes: 24'530'852)
RLE compresion info: compressed / uncompressed size / ratio: 52'160'044 
/ 78'078'696 / 34%
Chunk vector details: used 1'741'394 of 33'554'432 allocated entries (< 6%)

Thread worker-1 has finished
Thread worker-0 has finished
Thread worker-2 has finished
Time finished: Thu Nov 17 20:37:40 CET 2011
Total time taken: 1635s
20:37:40 europe eu 6550 this is run-10 starting to compile map with mkgmap

On 17.11.2011 19:30, Felix Hartmann wrote:
> Europe max-nodes 900.000 (resulting in 975 tiles, extract is about 4 
> weeks old already - hence 4 passes down to 1 pass )
>
> old splitter: 52min 38 seconds (3158 seconds)
> splitter_patched: 29min 45 seconds (1785 seconds)
> splitter patched with optimize-mem: 30min 41seconds (1841 seconds)
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111117/e6bc1b13/attachment.html 