[mkgmap-dev] Commit r4809: fix java.lang.AssertionError while building index from unicode tiles

Sun Oct 24 18:04:17 BST 2021

Hi Carlos & Gerd

Changing the default sort used when resources/sort/cp... matching the
code-page doesn't exist from cp1252 to cp65001/unicode stops this
crash. It probably gives a much better index. 

Except in the cases where the requested codepage uses transliterations
from resources/chars/ascii or latin1, I think the default sort should
be unicode. I haven't yet investigated how and when these
transliterations occur.

Tomorrow I'll look at reasons why the exception happens, even when the
sort is discarding significant characters.

Ticker

On Sun, 2021-10-24 at 18:31 +0200, Carlos Dávila wrote:
> The reason for using code-pages other than 65001 is that many Garmin 
> here: 
> https://openmtbmap.org/download/odbl/#Compatibility_-_Unicode_vs_Non_Unicode_cannot_authenticate_maps
> 
> El 24/10/21 a las 18:14, Ticker Berkin escribió:
> > Hi Carlos
> > 
> > When mkgmap doesn't have a resources/sort for the given code page,
> > it
> > defaults the sort to cp1252 (Western European).
> > 
> > As part of building the the various indexes, it sorts counties,
> > regions, cities, streets etc using this sort, but any characters
> > that
> > don't have a defined sort order are ignored in the ordering. The
> > result
> > of this is that, using cp1252 on Chinese, all names seem the same.
> > 
> > I suspect that indexes are mostly empty and find is ignoring them.
> > 
> > There is some logic that is differentiating the names in these
> > structures on exact naming, and this inconsistency causes the
> > assertion
> > crash.
> > 
> > The actual output in the map image is cp836, which Basecamp and
> > Mapsource appear to handle. I don't know how well it is supported
> > by
> > Garmin devices.
> > 
> > Is there a reason for using cp836 rather than cp65001/unicode?
> > 
> > Ticker
> > 
> > On Sun, 2021-10-24 at 16:22 +0200, Carlos Dávila wrote:
> > > using copy from JOSM/paste into BaseCamp, I could test address
> > > searches
> > > and they seem to work.
> > > 
> > > El 23/10/21 a las 23:50, Ticker Berkin escribió:
> > > > Hi Carlos
> > > > 
> > > > mkgmap doesn't have a resources/sort for code-page 936
> > > > (Microsoft's
> > > > character encoding for simplified Chinese). I was surprised it
> > > > doesn't
> > > > give any warning about this. I'll look more closely tomorrow to
> > > > see
> > > > what happens when it doesn't find the resource file.
> > > > 
> > > > I presume this didn't crash before, but did the index work?
> > > > 
> > > > I suspect this will have many of the same problems as unicode
> > > > sort
> > > > had
> > > > for unspecified characters.
> > > > 
> > > > I'll also investigate the other change relating to collation
> > > > strength.
> > > > 
> > > > Ticker
> > > > 
> > > > On Sat, 2021-10-23 at 22:26 +0200, Carlos Dávila wrote:
> > > > > Hi devs.
> > > > > 
> > > > > With this new version I get a new crash, but now with --code-
> > > > > page=936,
> > > > > not with unicode:
> > > > > 
> > > > > Exception in thread "main" java.lang.AssertionError: mdr20
> > > > > value
> > > > > changed
> > > > > f=5174 t=5180 count=2995
> > > > >            at
> > > > > uk.me.parabola.imgfmt.app.mdr.Mdr5Record.setMdr20(Mdr5Record.
> > > > > java
> > > > > :134
> > > > > )
> > > > >            at
> > > > > uk.me.parabola.imgfmt.app.mdr.Mdr20.buildFromStreets(Mdr20.ja
> > > > > va:8
> > > > > 4)
> > > > >            at
> > > > > uk.me.parabola.imgfmt.app.mdr.MDRFile.writeSections(MDRFile.j
> > > > > ava:
> > > > > 335)
> > > > >            at
> > > > > uk.me.parabola.imgfmt.app.mdr.MDRFile.write(MDRFile.java:270)
> > > > >            at
> > > > > uk.me.parabola.mkgmap.combiners.MdrBuilder.onFinish(MdrBuilde
> > > > > r.ja
> > > > > va:3
> > > > > 31)
> > > > >            at
> > > > > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:690)
> > > > >            at
> > > > > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsR
> > > > > eade
> > > > > r.ja
> > > > > va:126)
> > > > >            at
> > > > > uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:147)
> > > > >            at
> > > > > uk.me.parabola.mkgmap.main.Main.main(Main.java:118)
> > > > > 
> > > > > mkgmap command: java -ea -jar mkgmap-r4809.jar --index
> > > > > --bounds=bounds.zip --housenumbers --code-page=936
> > > > > 31177013.o5m
> > > > > 
> > > > > https://files.mkgmap.org.uk/download/524/31177013.o5m
> > > > > 
> > > > > El 22/10/21 a las 9:42, svn commit escribió:
> > > > > > Version mkgmap-r4809 was committed by gerd on Fri, 22 Oct
> > > > > > 2021
> > > > > > 
> > > > > > fix java.lang.AssertionError while building index from
> > > > > > unicode
> > > > > > tiles
> > > > > > mdrUnicode_v2.patch by Ticker Berkin
> > > > > > 
> > > > > > http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4809
> > > > > > _______________________________________________
> > > > > > mkgmap-dev mailing list
> > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > _______________________________________________
> > > > > mkgmap-dev mailing list
> > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > _______________________________________________
> > > mkgmap-dev mailing list
> > > mkgmap-dev at lists.mkgmap.org.uk
> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > 
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev