[mkgmap-dev] Exception in thread "main" java.lang.IllegalArgumentException . on Europe extract

Wed Dec 1 16:39:51 GMT 2021

Hi Felix,

your index is probably simply too large (> 2GB).
The attached patch fixes another possible overflow, but I am not sure if such a file size is allowed for the index.

Maybe try to remove specific items from the index (see --mdr7-excl, --mdr7-del)
I think the reason for the huge size is your style which combines a lot of information in the road name, e.g. "102 Thw-Göttinger Weg T2 Pth Zentralalpenweg 02 Hr"
or "137 Mindener Jubiläumsweg T2 Pth Mindener Jubiläumsweg Hr"
(found these in omtb_austria_13.05.2021)
All this information is stored in the PC index (a string table), and with --split-name-index lots of entries are produced for strings like "Pth" or "T2" or "Hr"

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver at gmail.com>
Gesendet: Mittwoch, 1. Dezember 2021 17:16
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Exception in thread "main" java.lang.IllegalArgumentException . on Europe extract

I found out that if I remove --split-name-index then it compiles fine. I'm just trying a minimal setup option wise and could look if it's a special .img tile or not.
I could upload all map tiles somewhere for further analysis - or try some more things before doing that (12GB data or so).

On Wed, 1 Dec 2021 at 13:08, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
Hi Gerd,

here is the log - hope you can find out something with it - but I think it is missing the problem - there are only 3 lines before the crash - all else is many minutes before.

2021/12/01 12:55:08 INFO (ImgFS): bs=512, whole size=-1789952000, hb=-14501, fb=-3481499, blocks=-3496000

2021/12/01 12:55:08 INFO (ImgFS): bs=1024, whole size=-1786235904, hb=-3623, fb=-1740748, blocks=-1744371

2021/12/01 12:55:08 INFO (ImgFS): Best block size: 512 sizeInBlocks=-3496000, reserved=-14501

On Wed, 1 Dec 2021 at 08:57, Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> wrote:
Hi Felix,

hm, in that case one of the two factors is already negative.
You can enable logging and run the combiner step again (no need to compile the tiles again)
Add this to the log config:
uk.me.parabola.imgfmt.sys.ImgFS.level=FINE

BTW: The index contains not only address search data, also POI.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
Gesendet: Mittwoch, 1. Dezember 2021 02:27
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Exception in thread "main" java.lang.IllegalArgumentException . on Europe extract

Hi Gerd,

thanks for the quick response. Any other way I can try to get more debug information?

I tried with the patch but I still get:
gmapi-minimal: Skipping file E:\OpenMTBMap\contourlines20\europe\75500448.img
Exception in thread "main" java.lang.IllegalArgumentException
        at java.base/sun.nio.ch<http://sun.nio.ch>.FileChannelImpl.position(FileChannelImpl.java:355)
        at uk.me.parabola.imgfmt.sys.ImgFS.sync(ImgFS.java:249)
        at uk.me.parabola.imgfmt.sys.ImgFS.close(ImgFS.java:325)
        at uk.me.parabola.mkgmap.combiners.MdrBuilder.onFinish(MdrBuilder.java:334)
        at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:690)
        at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:147)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:118)

On Tue, 30 Nov 2021 at 14:48, Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>> wrote:
Hi Felix,

might be an overflow. The code line is
file.position(param.getReservedDirectoryBlocks() * param.getBlockSize());

so two ints are multiplied.

Please try if the attached patch changes something.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:extremecarver at gmail.com>>>
Gesendet: Dienstag, 30. November 2021 13:23
An: Development list for mkgmap
Betreff: [mkgmap-dev] Exception in thread "main" java.lang.IllegalArgumentException . on Europe extract

Using mkgmap version 4808 - when compiling Europe. It went through fine on all geofabrik country extracts.
I then upgraded to mkgmap 4819 and yesterday downloaded a new europe extract (4 days newer) from geofabrik to see if I can reproduce it again - and the error message is identical.
I'm going to run with default style now again - to see if it makes a difference.

C:\openmtbmap\mkgmap.jar --max-jobs=11 --order-by-decreasing-area  "--generate-sea" --code-page=65001 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --lower-case --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13, 12:12" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6  --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4 --add-boundary-nodes-at-admin-boundaries=2 --cycle-map --ignore-fixme-values --housenumbers --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --link-pois-to-ways --ignore-turn-restrictions --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20
 , 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_eu --show-profiles=1  --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --country-abbr=eu --country-name=europe --mapname=65500000 --family-id=6550 --product-id=1 --series-name=omtb_europe_29.11.2021_UC_LOCAL --family-name=mtb_eu_29.11.2021_UC_LOCAL --tdbfile --x-gmapi-minimal --overview-mapname=mapsetc --keep-going --area-name="europe_29.11.2021_UC_LOCAL_omtb" -c D:\openmtbmap\maps\template.europe E:\OpenMTBMap\contourlines20\europe\7*.img typeu.TYP  1>NUL

Number of MapFailedExceptions: 0
gmapi-minimal: Writing freshly compiled file C:\openmtbmap\maps\.\65500000.img
gmapi-minimal: Writing freshly compiled file C:\openmtbmap\maps\.\65500001.img
gmapi-minimal: Writing freshly compiled file C:\openmtbmap\maps\.\65500002.img
gmapi-minimal: Writing freshly compiled file C:\openmtbmap\maps\.\65500003.img
gmapi-minimal: Writing freshly compiled file C:\openmtbmap\maps\.\65500004.img
................
skipped
.....................
long list of contourline files:
.........
gmapi-minimal: Skipping file E:\OpenMTBMap\contourlines20\europe\75500447.img
WARNING (global): Input file E:\OpenMTBMap\contourlines20\europe\75500448.img has different code page 1252
WARNING (global): Input file E:\OpenMTBMap\contourlines20\europe\75500448.img has different charset type 10
WARNING (global): Input files have different code pages
gmapi-minimal: Skipping file E:\OpenMTBMap\contourlines20\europe\75500448.img
.... 448 is the highest number contourlines file

............And here comes the error
it happens on writing the address index mdr file - mapset.img and mapset.tdb have already been created.

Exception in thread "main" java.lang.IllegalArgumentException
        at java.base/sun.nio.ch<http://sun.nio.ch><http://sun.nio.ch>.FileChannelImpl.position(FileChannelImpl.java:355)
        at uk.me.parabola.imgfmt.sys.ImgFS.sync(ImgFS.java:249)
        at uk.me.parabola.imgfmt.sys.ImgFS.close(ImgFS.java:325)
        at uk.me.parabola.mkgmap.combiners.MdrBuilder.onFinish(MdrBuilder.java:334)
        at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:690)
        at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:147)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:118)

--
Felix Hartman - Openmtbmap.org & VeloMap.org

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
Felix Hartman - Openmtbmap.org & VeloMap.org

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: overflow2.patch
Type: application/octet-stream
Size: 1009 bytes
Desc: overflow2.patch
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20211201/f700cb6b/attachment-0001.obj>