[mkgmap-dev] Error when running splitter with several input files, could there be keep-complete=single mode?

Mon Aug 30 11:50:36 BST 2021

Well I'm not fully sure this works as intended - though at first glance I
cannot find a problem - but running mkgmap on the split files gives an:

C:\openmtbmap\maps>start /belownormal /b /wait java -jar
-XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M
C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area
--code-page=1252 "--style-file=C:\openmtbmap\buildings_style"
 --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds
--ignore-turn-restrictions --merge-lines --allow-reverse-merge
--transparent --draw-priority=28 --add-pois-to-areas
--simplify-polygons=23:4,22:7,21:8
--copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
--license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
--add-boundary-nodes-at-admin-boundaries=2
--poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values
--polygon-size-limits="24:16, 23:14, 22:12, 21:11"
--ignore-turn-restrictions --description=buildings_fr --country-abbr=fr
--country-name=france --mapname=83910046 --family-id=8391 --product-id=1
--series-name="buildings_france_30.08.2021" --keep-going
--family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp
--overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c
D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ  1>NUL
Mkgmap version 4806M
Time started: Mon Aug 30 12:34:48 CEST 2021
Number of MapFailedExceptions: 0
WARNING (global): Could not copy 83910046.RGN
uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists
WARNING (global): Could not copy 83910046.TRE
uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists
WARNING (global): Could not copy 83910046.LBL
uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists
Number of ExitExceptions: 0
Time finished: Mon Aug 30 12:35:28 CEST 2021
Total time taken: 39 seconds

Now of course there has been no file at all called 83910046* in the
directory except 83910046.o5m (referenced via
D:\openmtbmap\maps\template.franceb1 input file option). Google spits out
no mention at all concerning this error. I would guess there is an error in
the --gmapi file?

On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver at gmail.com>
wrote:

> okay - well the only working solution to this problem is to use o5m output
> format with mkgmap, and then use osmconvert to join the files before
> splitting them again.
> Would it be possible for mkgmap to have a mode to split the files one by
> one without the detour via osmconvert?
>
> o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster
> with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m
> files vs osm.pbf - but small input files make not much time difference)...
>
> On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver at gmail.com>
> wrote:
>
>> I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start
>> /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m
>> C:\openmtbmap\splitter.jar --max-nodes=9600000  --max-threads=12
>> --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024
>> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
>> --description=france --mapid=83910000 france.osm  1>NUL
>>
>> Then I compiled the map using mgkamp.jar - and want to resplit all
>> remaining files that did not compile with smaller max-nodes value. However
>> splitter.jar chokes if I pass several files... Is there any way to have
>> splitter just working with several files without trying to merge them? I
>> just need the files split to the new max-nodes value and incrementing the
>> map id. If I pass only one file at a time, it is a PITA to script this (is
>> already hard enough figuring out the new map-id and creating a list of the
>> tiles).
>>
>> D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java
>> -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar
>> --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf
>> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
>> --description=france --mapid=83910046  "83910015.osm.pbf"
>> "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf"    1>NUL
>> Warning: --keep-complete is only used for the first input file. Further
>> files must use higher ids.
>> Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
>> This is not supported with keep-complete=true or --problem-list
>> uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted
>>         at
>> uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497)
>>         at
>> uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126)
>>         at
>> uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82)
>>         at
>> uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157)
>>         at
>> uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255)
>>         at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503)
>>         at uk.me.parabola.splitter.Main.start(Main.java:127)
>>         at uk.me.parabola.splitter.Main.main(Main.java:81)
>>
>>
>> The option to first join the files with osmconvert - is a PITA - as you
>> cannot parse several osm.pbf files to osmconvert (it only supports joining
>> osm or o5m files).
>> The option to supply splitter.jar with each file on it's own - is also a
>> PITA that would require horrendous for loops because after each split you
>> need to check which new map-id to use for the next file!
>>
>> So it would be great to have a keep-complete option that for each input
>> file, but not for all input files, uses keep-complete. Actually I think
>> this should be the default mode when supplying several input files. Even if
>> mkgmap actually cannot run this in parallel / multithreading it would be
>> way better than parsing each file on it's own...
>> Or am I dumbstruck in finding a way to resplit too big input files?
>>
>> --
>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>
>>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210830/dd1b8201/attachment-0001.html>