[mkgmap-dev] Using buffered io doubles splitter performance

Sat Nov 14 14:29:14 GMT 2009

I noticed that the splitter processing does a large number of small read
& writes. The speed can be doubled by using buffered IO as per the
attached patch.

Before:
$ time java -Xmx1500m -jar splitter/dist/splitter.jar --max-nodes=1000000 --cache=cache /store/planet/great_britain-20091114.osm.bz2
...
Wrote 11,101,332 nodes, 1,485,442 ways, 54,180 relations
Time finished: Sat Nov 14 13:43:40 GMT 2009
Total time taken: 675s

real    11m15.561s
user    6m17.064s 
sys     4m19.885s 

After:
Wrote 11,101,332 nodes, 1,485,442 ways, 54,180 relations
Time finished: Sat Nov 14 14:17:37 GMT 2009
Total time taken: 305s

real    5m5.343s
user    4m42.738s
sys     0m8.555s

I did remember to delete the cache between the two runs.

	Jon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: splitter-buffered-io.patch
Type: text/x-patch
Size: 3566 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20091114/1e3bc63c/attachment.bin 