[mkgmap-dev] Splitter only writes points, not ways on huge file?

Sat May 29 18:19:21 BST 2021

Hi Gerd,

thanks I assume there are no ways - and phyghtmap does not work correctly
either on such a huge area - has maybe a 64GB limit which would be broken?
or does not work on the poly file with two areas. I will run it again now
against both single areas (one has finished already and looks okay - but
that was the tiny area). The other area will need 15 days or so as
phyghtmap on ubuntu is using 1 thread only (can use 2 threads - but that is
only maybe 1% faster. if more threads just wasting memory but not being any
faster). So I will wait and see in 2 weeks...

osmconvert also needed like 60 minutes or so to put out:
timestamp min: 2021-05-13T08:41:54Z
timestamp max: 2021-05-13T08:41:54Z
lon min: 44.9999999
lon max: 179.9991666
lat min: -11.0025000
lat max: 81.8573611
nodes: 26761028462
ways: 0
relations: 0
node id min: 10000000
node id max: 26771065386

so clearly no ways.

On Sat, 29 May 2021 at 08:47, Gerd Petermann <
gpetermann_muenchen at hotmail.com> wrote:

> Hi Felix,
> the log shows
> "way Map is empty"
> so splitter doesn't find any ways in asia10m_.osm.pbf.
>
> So, either the file doesn't contain ways or splitter fails to find them.
> Maybe use
> osmconvert --out-statistics asia10m_.osm.pbf
> to find out more.
>
> Reg. performance when splitting huge files with --keep-complete
> use o5m format and --max-areas=4096 (or higher)
>
> I'll have to check why splitter needs so much time in the
> "Problem-list-generator" passes. Without ways or relations there is no need
> to even execute that.
>
> The asia.poly contains two rings. I think splitter can copy with that but
> maybe phyghtmap cannot? Try with two separate *.poly files
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
> Felix Hartmann <extremecarver at gmail.com>
> Gesendet: Freitag, 28. Mai 2021 19:29
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge
> file?
>
> Attached the full splitter.log - looks fine. Is it most likely that the
> error is on phyghtmap?
>
> On Fri, 28 May 2021 at 13:12, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> running now again with resolution 13, I had tried if changing resolution
> to 12 would solve the problem - but is does not.
>
> On Fri, 28 May 2021 at 13:11, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> Asia.poly is attached - I did not save the splitter.log - doing this right
> now (but will take some hours - splitting this huge file takes a lot of
> time)
>
> The log starts with:
> Splitter version 597 compiled 2020-01-30T14:10:47+0000
> boundary-tags=use-exclude-list
> cache=
> description=
> geonames-file=
> handle-element-version=remove
> ignore-osm-bounds=false
> keep-complete=true
> mapid=75400000
> max-areas=2048
> max-nodes=12000000
> max-threads=7
> mixed=false
> no-trim=true
> num-tiles=
> output=pbf
> output-dir=
> overlap=auto
> polygon-desc-file=
> polygon-file=
> precomp-sea=
> problem-file=
> problem-report=
> resolution=13
> route-rel-values=
> search-limit=200000
> split-file=
> status-freq=120
> stop-after=dist
> wanted-admin-level=5
> write-kml=
> Elapsed time: 0s   Memory: Current 1012MB (7MB used, 1005MB free) Max
> 25000MB
> Time started: Fri May 28 12:06:59 CEST 2021
> Map is being split for resolution 13:
>  - area boundaries are aligned to 0x800 map units (0.0439453125 degrees)
>  - areas are multiples of 0x800 map units wide and high
> Processing asia10m_.osm.pbf
> Bounding box -179.99990000000003 -13.011650000000001 179.99990000000003
> 84.52666
> 10,000,000 nodes parsed... id=19999999
> 20,000,000 nodes parsed... id=29999999
> 30,000,000 nodes parsed... id=39999999
> 40,000,000 nodes parsed... id=49999999
> 50,000,000 nodes parsed... id=59999999
> 60,000,000 nodes parsed... id=69999999
> 70,000,000 nodes parsed... id=79999999
> 80,000,000 nodes parsed... id=89999999
> 90,000,000 nodes parsed... id=99999999
> 100,000,000 nodes parsed... id=109999999
>
> On Fri, 28 May 2021 at 12:25, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
> wrote:
> Hi Felix,
>
> the pbf file really contains only nodes.
> Can you attach the splitter log file and the asia.poly, please?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
> Gesendet: Freitag, 28. Mai 2021 11:18
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge
> file?
>
> Hi Gerd,
> sorry - fixed. I attached the file. I could also put the whole 60GB file
> for analysis. But do not know if you could download such a huge file or
> analyse it better.
> https://openmtbmap.org/75400407.osm.pbf
>
> Felix
>
> On Fri, 28 May 2021 at 09:03, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com
> ><mailto:gpetermann_muenchen at hotmail.com<mailto:
> gpetermann_muenchen at hotmail.com>>> wrote:
> Hi Felix,
>
> the link to the pbf file seems to be wrong?  I get an 404
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto:
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>>
> Gesendet: Freitag, 28. Mai 2021 07:56
> An: Development list for mkgmap
> Betreff: [mkgmap-dev] Splitter only writes points, not ways on huge file?
>
> I used phyghtmap to create a huge file (60GB) of 10m equidistance osm.pbf
> contourlines data.
>
> phyghtmap --jobs=2 --polygon=/home/contourlines/bounds/asia.poly
> --step=10  --no-zero-contour --void-range-max=-420
> --output-prefix="Asia10m" --line-cat=500,100--start-node-id=10000000
> --start-way-id=10000000 --source=view1,srtm1,view3,srtm3
> --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf
>
> I am not fully sure that this pbf file is correct, but I created it
> exactly the same way as smaller countries. Maybe its also phyghtmap going
> wrong. But I doubt it with phyghtmaps incremental approach of just
> constantly writing out data.
>
> When splitting it with splitter - and opening up one of the osm.pbf files
> in JOSM - I noticed it contains only the contourline nodes, no lines.
> I think the Asia continent file spans 360 degrees - so maybe that causes
> problems with splitter?
>
>
>
> I have uploaded a splitted file here:
> https://openmtbmap.org/75400407.osm.pbf
> If you load it in Josm, you will notice it is alll the points but no
> lines...
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
> mkgmap-dev at lists.mkgmap.org.uk>>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210529/42123341/attachment.html>