[mkgmap-dev] Osmium or Splitter/mkgmap issue?

Thu Jun 11 22:31:19 BST 2020

Gerd thanks for the script. I'll give a try. Anyway, when using osmium
merge on   areas bigger than Malta (Italy + countries around Alps) it works
fine and the final img file is the expected one.

On the other side, you wrote:
> maybe the two files 70000001.osm.pbf have different bounds?

and you hit it!

*# Using Osmosis*
enrico at gling:/opt/osm/tmp$ ../osmosis/bin/osmosis --rbf
./malta-latest.osm.pbf --rbf Contours-Malta.pbf --merge --wb all.pbf
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar
--mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 23:19:43 CEST 2020
#
70000001: 1658880,649216 to 1693696,694272
#       : *35.595703,13.930664 to 36.342773,14.897461*

*#Using osmium sort*
enrico at gling:/opt/osm/tmp$ osmium sort *.pbf -o all.pbf
[======================================================================]
100%
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar
--mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 22:59:26 CEST 2020
#
70000001: 1658880,649216 to 1693696,694272
#       : *35.595703,13.930664 to 36.342773,14.897461*

*# While when using "osmium merge"...*
enrico at gling:/opt/osm/tmp$ osmium merge *.pbf -o all.pbf
[======================================================================]
100%
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar
--mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 23:17:46 CEST 2020
#
70000001: 1454080,247808 to 2002944,1394688
#       : *31.201172,5.317383 to 42.978516,29.926758*

and thus the latter produces a much bigger img!

Thus question is...is osmium merge that creates a much bigger area? Or is
it splitter that gets confused by something?

Attached the logs, around line 40 you can see splitters mentions "Bounding
box 13.815377000000002 35.519850000000005 14.856099 36.332958000000005"
only when using osmosis or osmium sort.

If anyone is interested to debug this further I can provide the pbf files
produced by osmium&osmosis.

Enrico

On Thu, Jun 11, 2020 at 6:50 AM Gerd Petermann <
gpetermann_muenchen at hotmail.com> wrote:

> Hi Enrico,
>
> I use osmconvert like this in a script:
> set input=f:\osm\niedersachsen-latest.osm.pbf
> ...
> rmdir /s/q split
> if not exist part.o5m f:\osm\osmconvert --drop-author -B=map.poly %input%
> -o=part.o5m
> if not exist srtm.o5m f:\osm\osmconvert --drop-author -B=map.poly
> f:\osm\Hoehendaten_Freizeitkarte_EUROPE.osm.pbf -o=srtm.o5m
> rem merge
> if not exist work.o5m f:\osm\osmconvert part.o5m srtm.o5m -o=work.o5m
> java -Xmx4G -jar d:\splitter\dist\splitter.jar --output-dir=split
> --max-nodes=1000000 --mapid=%mapid%0001 --write-kml=splitter.kml work.o5m
> > splitter_%dt%-%t%.log
>
> I create the file map.poly with JOSM (and the poly plugin). Instead of
> niedersachsen-latest.osm.pbf I can also use a larger file like
> europe-latest.osm.pbf if needed.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
> Enrico Liboni <eliboni at gmail.com>
> Gesendet: Mittwoch, 10. Juni 2020 23:52
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Osmium or Splitter/mkgmap issue?
>
> Acth...
>
> * osmium sort "...will  take  roughly  10 times as much memory as the
> files take on disk in  osm.pbf format" - thus I can't pursue this  option
> when dealing with several countries :(
> * osmconvert seems to need o5m, not pbf... I have all pbfs.
>
>
>
> On Wed, Jun 10, 2020 at 10:56 PM Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
> wrote:
> Hi Enrico,
>
> I use osmconvert. I am not familar with osmosis and I cannot use osmium on
> Windows.
> sort instead of merge sounds good, but I would expect an error message
> from splitter when data is not sorted correctly.
> You should also check the output from splitter reg. the bounds.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Enrico Liboni <
> eliboni at gmail.com<mailto:eliboni at gmail.com>>
> Gesendet: Mittwoch, 10. Juni 2020 22:49
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Osmium or Splitter/mkgmap issue?
>
> Gerd -  however I see no reason why bounds should be different, I use the
> very same ones. By the way, as per my other email, it seems that usong
> osmium sort instead of osmium merge does the trick. Perhaps objects are not
> sorted as osmium merge expects...
>
> What do you usually use to merge pbfs?
>
> Thanks!
> Enrico
>
> On Wed, Jun 10, 2020 at 10:27 PM Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com
> ><mailto:gpetermann_muenchen at hotmail.com<mailto:
> gpetermann_muenchen at hotmail.com>>> wrote:
> Hi Enrico,
>
> maybe the two files 70000001.osm.pbf have different bounds? If one is much
> larger the difference could be the additional data for sea and background
> polygons.
> In one command you list the input files, in the other you use *.pbf.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Enrico Liboni <
> eliboni at gmail.com<mailto:eliboni at gmail.com><mailto:eliboni at gmail.com
> <mailto:eliboni at gmail.com>>>
> Gesendet: Mittwoch, 10. Juni 2020 22:19
> An: Development list for mkgmap
> Betreff: [mkgmap-dev] Osmium or Splitter/mkgmap issue?
>
> I'm getting a weird behaviour: I merge <5MB pbfs,  when using osmium I get
> a 8MB img file while with osmosis it is less than 4MB! The latter seems
> fine since the initial pbfs are less than 5MB. I'd like to use osmium in my
> scripts since it performs better.
>
> Am I doing something wrong? Thanks to anyone that could shed some light on
> this!
>
> # input pbfs
> -r--r-----  1 enrico enrico 4673440 Jun 10 21:50 malta-latest.osm.pbf
> -r--r-----  1 enrico enrico   15376 Jun 10 21:50
> Malta_lon14.03_14.74lat35.65_36.00_view3.osm.pbf
> -r--r-----  1 enrico enrico    8300 Jun 10 21:50
> Malta_lon14.03_14.74lat36.00_36.18_view3.osm.pbf
> # using osmium
> $ osmium merge *.pbf -o all.pbf
> $ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
> $ java -jar ../mkgmap/mkgmap.jar  --family-id=10030 --product-id=1
> --route --remove-short-arcs  --bounds=../bounds.zip \
>  --precomp-sea=../sea.zip  --location-autofill=is_in,nearest
> --draw-priority=20 --gmapsupp  --index --housenumbers 7000*pbf
>
> -rw-rw-r-- 1 enrico enrico 4695873 Jun 10 21:54 all.pbf
> -rw-rw-r-- 1 enrico enrico 4288669 Jun 10 21:54 70000001.osm.pbf
> -rw-rw-r-- 1 enrico enrico 7925760 Jun 10 21:55 70000001.img
> -rw-rw-r-- 1 enrico enrico 8171520 Jun 10 21:55 gmapsupp.img
>
> # using osmosis
> $ ../osmosis/bin/osmosis --rbf ./malta-latest.osm.pbf \
>  --rbf ./Malta_lon14.03_14.74lat35.65_36.00_view3.osm.pbf \
>  --rbf ./Malta_lon14.03_14.74lat36.00_36.18_view3.osm.pbf \
>  --merge --merge --wb all.pbf
> $ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
> $ java -jar ../mkgmap/mkgmap.jar  --family-id=10030 --product-id=1
> --route --remove-short-arcs  --bounds=../bounds.zip \
>  --precomp-sea=../sea.zip  --location-autofill=is_in,nearest
> --draw-priority=20 --gmapsupp  --index --housenumbers 7000*pbf
>
> -rw-rw-r--  1 enrico enrico 4680030 Jun 10 22:03 all.pbf
> -rw-rw-r--  1 enrico enrico 4288668 Jun 10 22:04 70000001.osm.pbf
> -rw-rw-r--  1 enrico enrico 3547136 Jun 10 22:04 70000001.img
> -rw-rw-r--  1 enrico enrico 3788800 Jun 10 22:04 gmapsupp.img
>
> Using very latest splitter, mkgmap, and osmium (tried with 1.10 and 1.12
> compiled from source...).
> Note both gmapsupp.img seems to work just fine on the garmin device.
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
> mkgmap-dev at lists.mkgmap.org.uk>>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20200611/b1f0e44b/attachment-0001.html>
-------------- next part --------------
enrico at gling:/opt/osm/tmp$ osmium merge *.pbf -o all.pbf
[======================================================================] 100% 
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
Splitter version 597 compiled 2020-01-30T14:10:47+0000
boundary-tags=use-exclude-list
cache=
description=
geonames-file=
handle-element-version=remove
ignore-osm-bounds=false
keep-complete=true
mapid=70000001
max-areas=2048
max-nodes=1600000
max-threads=4 (auto)
mixed=false
no-trim=false
num-tiles=
output=pbf
output-dir=
overlap=auto
polygon-desc-file=
polygon-file=
precomp-sea=
problem-file=
problem-report=
resolution=13
route-rel-values=
search-limit=200000
split-file=
status-freq=120
stop-after=dist
wanted-admin-level=5
write-kml=
Elapsed time: 0s   Memory: Current 115MB (5MB used, 110MB free) Max 1700MB
Time started: Thu Jun 11 22:58:36 CEST 2020
Map is being split for resolution 13:
 - area boundaries are aligned to 0x800 map units (0.0439453125 degrees)
 - areas are multiples of 0x800 map units wide and high
Processing all.pbf
Fill-densities-map pass took 367 ms
Exact map coverage read from input file(s) is (31.209990978240967,5.3397417068481445) to (42.94660806655884,29.90999937057495)
Rounded map coverage is (31.201171875,5.3173828125) to (42.978515625,29.9267578125)
Splitting nodes into areas containing a maximum of 1,600,000 nodes each...
Highest node count in a single grid element is 41,049
Solving partition (31.201171875,5.3173828125) to (42.978515625,29.9267578125) with 426,742 nodes
Trying to find nice split for (31.201171875,5.3173828125) to (42.978515625,29.9267578125) with 426,742 nodes
searching for split with min-nodes 80000, learned 0 good partial solutions
Best solution until now: 1 tile(s). The smallest node count is 426742 (26 %), the worst aspect ratio is near 1.79, elapsed search time: 0 s
This can't be improved.
Solution is nice. Can't find a better solution with search-limit 200000: 1 tile(s). The smallest node count is 426742 (26 %), the worst aspect ratio is near 1.79
Final solution: 1 tile(s). The smallest node count is 426742 (26 %), the worst aspect ratio is near 1.79
This seems to be nice.
Area 70000001 covers (31.201171875,5.3173828125) to (42.978515625,29.9267578125) and contains 426742 nodes (26 %)
Creating the initial areas took 17 ms
1 areas:
Area 70000001: 1454080,247808 to 2002944,1394688 covers (0x163000,0x3c800) to (0x1e9000,0x154800)
Generating problem list for 1 distinct areas
Processing 1 areas in a single pass
Pseudo areas:
Pseudo area -2 covers (42.978515625,-180.0) to (90.0,180.0)
Pseudo area -3 covers (-90.0,-180.0) to (31.201171875,180.0)
Pseudo area -4 covers (31.201171875,-180.0) to (42.978515625,5.3173828125)
Pseudo area -5 covers (31.201171875,29.9267578125) to (42.978515625,180.0)
cached 6 combinations of areas that form rectangles.
AreaGridTree [512][512] for grid area (-90.0,-180.0) to (90.0,180.0) requires max. 3 checks for each node (0 sub grid(s))
Starting problem-list-generator pass(es)
-----------------------------------
Starting problem-list-generator pass 1 of 1
way Map: uses SparseLong2IntMap
coord Map: uses SparseLong2IntMap
Processing all.pbf
Processing all.pbf
coord Map: 426,742 stored long/int pairs require ca. 1141 bytes per pair. 32,540 chunks are used, the avg. number of values in one 64-chunk is 13.
coord Map details: ~465 MB, including 58 array(s) with 8 MB

way Map: 65,454 stored long/int pairs require ca. 1027 bytes per pair. 11,456 chunks are used, the avg. number of values in one 64-chunk is 5.
way Map details: ~64 MB, including 8 array(s) with 8 MB

  Number of stored area combis for nodes: 426,742
  Number of stored area combis for ways: 65,454
  Number of stored Integers for rels: 0
  Number of stored combis in dictionary: 17
  Number of detected problem ways: 0
  Number of detected problem rels: 0
  JVM Memory Info: Current 1109MB (621MB used, 488MB free) Max 1700MB

Problem-list-generator pass 1 took 533 ms
Problem-list-generator pass(es) took 596 ms
cached 0 combinations of areas that form rectangles.
AreaGridTree [512][512] for grid area (31.201171875,5.3173828125) to (42.978515625,29.9267578125) requires max. 1 checks for each node (0 sub grid(s))
Writing temp files took 0 ms
Distributing data Thu Jun 11 22:58:37 CEST 2020
Processing 1 areas in a single pass
coord Map: uses SparseLong2IntMap
way Map: uses SparseLong2IntMap
Starting distribution pass 1 of 1, processing 1 areas (70000001 to 70000001)
Processing all.pbf
Writing ways Thu Jun 11 22:58:38 CEST 2020
Writing relations Thu Jun 11 22:58:38 CEST 2020
coord Map: 426,742 stored long/int pairs require ca. 1141 bytes per pair. 32,540 chunks are used, the avg. number of values in one 64-chunk is 13.
coord Map details: ~465 MB, including 58 array(s) with 8 MB

way Map: 65,454 stored long/int pairs require ca. 1027 bytes per pair. 11,456 chunks are used, the avg. number of values in one 64-chunk is 5.
way Map details: ~64 MB, including 8 array(s) with 8 MB

  JVM Memory Info: Current 1153MB (579MB used, 574MB free) Max 1700MB
Full Node tests:  2
Quick Node tests: 426,740
Thread worker-0 has finished
Distribution pass(es) took 1376 ms
Time finished: Thu Jun 11 22:58:38 CEST 2020
Total time taken: 2 seconds
enrico at gling:/opt/osm/tmp$ cat areas.poly 
area
1
  5.317383  31.201172
  5.317383  42.978516
  29.926758  42.978516
  29.926758  31.201172
  5.317383  31.201172
END
END
enrico at gling:/opt/osm/tmp$ 