[mkgmap-dev] DEM File Format and mkgmap

Sun Dec 3 11:16:10 GMT 2017

Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile. 
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format: 
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value 
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are 
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works. 
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
        at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
        at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
        at test.display.DemDisplay.print(DemDisplay.java:845)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd