[mkgmap-dev] Memory limits for mkgmap and splitterFrom Chris Miller chris.miller at kbcfp.com on Fri Aug 7 16:13:46 BST 2009
bz2 is *very* slow to decompress, so yes if you have the space I'd recommend decompressing the osm first before running the splitter (since the splitter has to make a minimum of two passes over the file, thus also decompressing it at least twice). The (limited and simple) benchmarks I tried with .bz2 vs .osm showed that .bz2 splitting takes ~6 times longer than an uncompressed .osm file. As for gz - it is quite a lot faster to deal with than bz2 though I haven't done any benchmarking with it as far as the splitter is concerned. My guess is that uncompressed will still win out unless you have fairly slow disks and a very fast CPU. Interestingly, it's theoretically possible to parallelise bz2 compression/decompression algorithm to give an almost linear performance improvements per core. Implementing this would be a big job but on a 4+ core machine would make a pretty significant difference. It's on my todo list but please don't hold your breath! Chris > Just out of interest, what performance gains (or disadvantages) would > there be to working with uncompressed files, instead of bz2 and gz > files? > > Would this be faster for those of us with copious amounts of disk > space, or would the extra IO negate any CPU-related performance gains? > > I know that Osmosis performance on multi-core systems can apparently > be improved by piping the OSM file through a decompression program, > but I assume that would not be practical for Splitter which must make > several passes through the file. > > Cheers.
- Previous message: [mkgmap-dev] Memory limits for mkgmap and splitter
- Next message: [mkgmap-dev] Memory limits for mkgmap and splitter
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list