[mkgmap-dev] Error when running splitter with several input files, could there be keep-complete=single mode?

Mon Aug 30 11:38:19 BST 2021

okay - well the only working solution to this problem is to use o5m output
format with mkgmap, and then use osmconvert to join the files before
splitting them again.
Would it be possible for mkgmap to have a mode to split the files one by
one without the detour via osmconvert?

o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster
with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m
files vs osm.pbf - but small input files make not much time difference)...

On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver at gmail.com>
wrote:

> I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start
> /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m
> C:\openmtbmap\splitter.jar --max-nodes=9600000  --max-threads=12
> --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024
> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
> --description=france --mapid=83910000 france.osm  1>NUL
>
> Then I compiled the map using mgkamp.jar - and want to resplit all
> remaining files that did not compile with smaller max-nodes value. However
> splitter.jar chokes if I pass several files... Is there any way to have
> splitter just working with several files without trying to merge them? I
> just need the files split to the new max-nodes value and incrementing the
> map id. If I pass only one file at a time, it is a PITA to script this (is
> already hard enough figuring out the new map-id and creating a list of the
> tiles).
>
> D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java
> -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar
> --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf
> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
> --description=france --mapid=83910046  "83910015.osm.pbf"
> "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf"    1>NUL
> Warning: --keep-complete is only used for the first input file. Further
> files must use higher ids.
> Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
> This is not supported with keep-complete=true or --problem-list
> uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted
>         at
> uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497)
>         at
> uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126)
>         at
> uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82)
>         at
> uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157)
>         at
> uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255)
>         at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503)
>         at uk.me.parabola.splitter.Main.start(Main.java:127)
>         at uk.me.parabola.splitter.Main.main(Main.java:81)
>
>
> The option to first join the files with osmconvert - is a PITA - as you
> cannot parse several osm.pbf files to osmconvert (it only supports joining
> osm or o5m files).
> The option to supply splitter.jar with each file on it's own - is also a
> PITA that would require horrendous for loops because after each split you
> need to check which new map-id to use for the next file!
>
> So it would be great to have a keep-complete option that for each input
> file, but not for all input files, uses keep-complete. Actually I think
> this should be the default mode when supplying several input files. Even if
> mkgmap actually cannot run this in parallel / multithreading it would be
> way better than parsing each file on it's own...
> Or am I dumbstruck in finding a way to resplit too big input files?
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210830/00bbe57e/attachment.html>