[mkgmap-dev] Slowness processing specific tile

Sun May 22 21:24:12 BST 2011

Okay, some more tries to find out why mkgmap locks up on iceland (but 
not on most countries, so the thing is still really strange):

I'll upload all needed files to recreate the problem in another mail, 
cause that will be easier than e-mailing forth/back to find out why the 
problem happens.

First using my normal commandline:
start /low /b /wait java -ea -jar -Xmx6500M c:\openmtbmap\mkgmap.jar 
%style-file% --max-jobs=4 %generate-sea% --reduce-point-density=4 --nsis 
--index --transparent --adjust-turn-headings --add-pois-to-areas 
--ignore-maxspeeds --x-reduce-point-density-polygon=8 
--link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 
--remove-short-arcs=4 --description=openmtbmap_%abr% --merge-lines 
--location-autofill=1 --route --country-abbr=%abr% 
--country-name=%country% %profile% --mapname=%FID%0000 --family-id=%FID% 
--product-id=1 --series-name=openmtbmap_%country%_%date% 
--family-name=mtbmap_%abr%_%date% --tdbfile --overview-mapname=mapset 
--keep-going --area-name="%country%_%date%_openmtbmap.org" -c 
c:\openmtbmap\maps\template.%country%

Country is Iceland. I hope the error message on 2. can provide the 
needed information why the locator branch gets stuck indefinitely??
Note, I'm not using pbf as input files, but standard osm.gz, therefore I 
find it strange that including pbf support without any other changes, 
creates an Fatal Error, plus adds about 18% of time surplus to render 
the map.

I clearly have a problem with a lib, but have no clue how to solve this. 
Therefore please, include the libs into mkgmap svn, everything else will 
cause faults. I clearly have the libs that mkgmap wants, but probably 
got a wrong version that mkgmap does not expect at some point. However 
also the mkgmap_locator_r1492.jar downloaded from mkgmap.org.uk gets 
stuck for me indefinitely, so there definitely is a problem.

Here is the output on building mkgmap.jar, it looks fine to me:
d:\Garmin\mkgmap_svn_trunk>start /b /wait ant dist
Buildfile: build.xml

prepare:
     [mkdir] Created dir: d:\Garmin\mkgmap_svn_trunk\build\classes
compile:
     [javac] Compiling 378 source files to 
d:\Garmin\mkgmap_svn_trunk\build\classes
     [javac] Note: Some input files use unchecked or unsafe operations.
     [javac] Note: Recompile with -Xlint:unchecked for details.
compile-pbf:
      [echo] Protobuf binary format support
     [javac] Compiling 3 source files to 
d:\Garmin\mkgmap_svn_trunk\build\classes
build:
      [copy] Copying 422 files to d:\Garmin\mkgmap_svn_trunk\build\classes
dist:
       [jar] Building jar: d:\Garmin\mkgmap_svn_trunk\dist\mkgmap.jar

*1. mkgmap_trunk without pbf support (including my patches):* 44 seconds
No error message.

*2. mkgmap_trunk with pbf support (including my patches)*: 50 seconds 
plus error message (but map created):
[Fatal Error] :66:2: The content of elements must consist of well-formed 
character data or markup.
org.xml.sax.SAXParseException: The content of elements must consist of 
well-formed character data or markup.
         at 
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
         at 
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
         at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
         at 
uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70)
         at 
uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47)
         at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68)
         at 
uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103)
         at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:97)
         at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:65)
         at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:224)
         at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:221)
         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
         at java.util.concurrent.FutureTask.run(Unknown Source)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)
         at java.lang.Thread.run(Unknown Source)
[Fatal Error] :66:2: The content of elements must consist of well-formed 
character data or markup.
org.xml.sax.SAXParseException: The content of elements must consist of 
well-formed character data or markup.
         at 
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
         at 
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
         at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
         at 
uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70)
         at 
uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47)
         at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68)
         at 
uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103)
         at 
uk.me.parabola.mkgmap.combiners.TdbBuilder.writeOverviewMap(TdbBuilder.java:246)
         at 
uk.me.parabola.mkgmap.combiners.TdbBuilder.onFinish(TdbBuilder.java:239)
         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:419)
         at 
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
         at uk.me.parabola.mkgmap.main.Main.main(Main.java:129)

2a) as above but using mkgmap.jar trunk rev 1949 as downloaded from 
http://www.mkgmap.org.uk/snapshots/mkgmap-r1949.zip
No errors, but also I don't know if pbf support is icluded here??

*3. mkgmap_locator with my patches including pbf support (without using 
bound option with newest bound files as published here on the list):* No 
map created but 0bit file only, and mkgmap stuck for 20minutes.
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Element lists created after 627 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Quadtree created after 3194 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Location hook finished in 3823 ms

*4. mkgmap_locator as above, but adding 
--createboundsfile=c:\openmtbmap\maps\boundaries" to the input options:* 
No map created but 0bit file only, and mkgmap stuck for 20minutes.
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Element lists created after 657 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Quadtree created after 3021 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Location hook finished in 3569 ms

*5. Same as 3. But using --max-jobs=1:* No map created, but 0bit file 
only, mkgmap stuck for 20 minutes.
21:55:40 iceland is 6399 this is run6
start compilation 21:55:52 iceland
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Element lists created after 350 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Quadtree created after 1660 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Location hook finished in 2277 ms

*6. Same as 3. but using mkgmap_locator_r1952 downloaded from 
mkgmap.co.uk.:* mkgmap stuck for 20 minutes .... no map created, only 
0bit file.
22:11:48 iceland is 6399 this is run6
start compilation 22:12:00 iceland
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Element lists created after 232 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Quadtree created after 2587 ms
SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: 
Location hook finished in 3158 ms

On 20.05.2011 16:00, Nakor Osm wrote:
> * also be too low if you use big tiles and a high number
>
>     > of threads.
>     Well I use small tiles (max-nodes 900.000 on Turkey) plus only 4
>     threads. (I could use 8 due to Hyperthreading, but kept it at 4).
>     I even
>     have this problem on single tiles where the osm.gz is like 10MB. There
>     shouldn't be a slow down due to memory in that case, should there? (I
>     cannot check for memory use, as my user rights are too low).
>
>
> What happens if you only use one thread?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20110522/54570c6e/attachment.html 