[mkgmap-dev] Problem with splitter: Failed to find a correct split
From Patrik Brunner patrik.brunner at gmx.net on Fri Mar 25 12:32:24 GMT 2016
Good news, Gerd. .. as long as it's not too much thoughts... ;-) Just to give you a heads up regarding the 'source' of the problem: according to GeoFabrik the wrong bounds tag seems to be caused by osmium-history-splitter used to split the world file. Cheers Patrik On 25.03.2016 11:00, Gerd Petermann wrote: > > Hi Patrik, > > > reg. performance for option 3: I don't expect any extra costs for > this, splitter already collects all the data and applies the bbox > > after that. So, it always creates a (virtual) grid for the whole > planet. With normal resolutions this never was a bottle neck. > > > reg. implementation: I don't see any problem for any of the three > options, but option 3 may requrie more thoughts. > > > Gerd > > > > ------------------------------------------------------------------------ > *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag > von Patrik Brunner <patrik.brunner at gmx.net> > *Gesendet:* Freitag, 25. März 2016 10:35 > *An:* mkgmap-dev at lists.mkgmap.org.uk > *Betreff:* Re: [mkgmap-dev] Problem with splitter: Failed to find a > correct split > Gerd, > > yes, it's a complicated world with these non-rectancular countries, > can't we change that ? ;-) > > I did some tests yesterday playing around with --no-trim and > --polygon-files with Luxembourge (small enough to do quick tests), the > result is not that much different if I'm skipping --no-trim or use the > polygon file, the result ist still 'quite' rectancular, just a few > bits are really left aside.... but I will have to do more tests with > other countries (more complicated ones and bigger ones than LUX) to > see if in my case I could/should skip --no-trim for building other > maps too. > > Regarding your implementation option: > > * I would rather prefer not having option 2) implemented for the > same reason as you mention: it changes the default behaviour for > everyone. And as we have to say that for (rough guess) 99% of the > cases the actual behaviour is working that's too much. > * I like option 3). But enabling that by default would possibly > increase the time used to split as it first has to run through the > file to see which bounds (bounds tag or calculated one) have to be > used... but I'm sure you could answer that much better than me... ;-) > * Option 1) is possibly the cheapest to implement ? > > Thanks Patrik > > On 25.03.2016 07:33, Gerd Petermann wrote: >> >> Hi Patrik, >> >> >> good question. I also wondered if splitter can be changed to ignore >> wrong bounds >> >> tags. My thoughts: >> >> In your case the bbox is far too large, means it claims to contain >> data that is not there. >> >> The normal case is vice versa: The file contains some nodes outside >> of the bbox, >> >> maybe from ferry lines or other long ways, and my understanding is >> that we don't >> >> want to calculate the tiles based on those few nodes, since we might >> get a map with >> >> larger empty areas at the "border". No idea if that would cause more >> trouble. >> >> With typical data from geofabrik and the --no-trim option there will >> always be large >> >> empty areas as most countries are not rectangular ;-) >> >> >> Possible changes in the code: >> >> 1) add a --ignore-osm-bounds option and set the default to false >> (means one gets >> >> the same result like now when he uses the default) >> >> 2) add a --use-osm-bounds option and set the default to false >> >> (means one might get a different result when using the default) >> >> 3) add code to check how the collected data matches the given bounds, >> >> use whatever is smaller. This might also be triggered by an option if >> needed. >> >> >> I'd like to get some feedback from others first. >> >> >> Gerd >> >> >> >> ------------------------------------------------------------------------ >> *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag >> von KeenOnKites <keenonkites at gmx.net> >> *Gesendet:* Freitag, 25. März 2016 00:13 >> *An:* mkgmap-dev at lists.mkgmap.org.uk >> *Betreff:* Re: [mkgmap-dev] Problem with splitter: Failed to find a >> correct split >> Gerd, >> >> I couldn't go to sleep without getting to the ground of this... >> >> I performed the following tests on linux, always with my standard >> --no-trim present: >> >> * original osm.pbf file downloaded from geofabrik (with the >> problematic bounds tag in it) >> -> FAILURE >> * converted with the help of osmconvert to normal osm format >> -> FAILURE (which was to be expected, same file, just different >> format) >> * deleted the osm tag with a simple command like 'cat file.osm | >> grep -v bounds > file-nobounds.osm' and ran splitter against it, >> always with the same options >> -> SUCCESS >> >> This leads me to the question why the bounds tag is read/used if it >> also works without and even gives less problem ? >> >> >> .... sorry for bombarding you (and the list), but I think it's >> nevertheless and interesting question. >> >> >> Cheers >> Patrik >> >> >> On 24.03.2016 23:51, KeenOnKites wrote: >>> Gerd, >>> >>> I did dig a bit deeper... also as it rang a bell: >>> we had quite a similar problem with the wrong bounding box in Alaska >>> already October 2014. It was an illegal value of maxlon="180.0005" >>> causing splitter to bail out >>> >>> When I convert the osm.pbf file with the help of osmconvert to osm >>> and look at the first few lines I see a 'bounds' tag announcing the >>> problematic bounding box (not illegal as in 2014, but still >>> 'suboptimal'): >>> >>> <?xml version='1.0' encoding='UTF-8'?> >>> <osm version="0.6" generator="osmconvert 0.8.2" >>> timestamp="2016-03-23T20:13:02Z"> >>> <bounds minlat="49.8089" minlon="-179.9532" >>> maxlat="73.79794" maxlon="179.9999"/> >>> <node id="27207079" lat="64.7487541" lon="-147.3242821" >>> version="2" >>> ... >>> >>> >>> Getting the statistics via osmconvert with --out-statistics seems to >>> read through the file and checks the 'real' bounding box: >>> >>> ... >>> lon min: -180.0000000 >>> lon max: -122.5122525 >>> lat min: 48.6234931 >>> lat max: 71.6061501 >>> ... >>> >>> >>> I'm now wondering if splitter get's confused by the existing but >>> obviously strange bounds tag. >>> >>> According to the findings in 2014 splitter can handle files without >>> the bounds tag and just gets the real boundings from all the >>> elements in the file. >>> I did not test to run it through splitter without the bounds tag as >>> I'm having troubles converting the file properly on my windows... >>> but I'll try that probably tomorrow again on linux (sort of late >>> already today). >>> The process would be >>> >>> osm.pbf -> osm -> get rid of the bounds tag -> back to osm.pbf >>> again (to have same source file format) >>> .... and then run the splitter and see what happens. >>> >>> But if I have a proper osm file with the 'problematic' bounds tag in >>> it, I can also try to reproduce the problem with the osm file. If it >>> is reproducible I just drop the tag and try it again. >>> >>> BTW: I've contacted geofabrik already via email >>> >>> Patrik >>> >>> >>> On 24.03.2016 22:27, KeenOnKites wrote: >>>> Gerd, >>>> >>>> I'll play a bit more with this option and check what suits me best. >>>> >>>> Again thanks for the incredibly quick answer to my problem/question. >>>> >>>> Cheers >>>> Patrik >>>> >>>> On 24.03.2016 22:20, Gerd Petermann wrote: >>>>> >>>>> Hi Patrik, >>>>> >>>>> >>>>> I think the wanted effect of the no-trim option is a rectangular map, >>>>> >>>>> which some people prefer, esp. on the PC. >>>>> >>>>> >>>>> Gerd >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im >>>>> Auftrag von KeenOnKites <keenonkites at gmx.net> >>>>> *Gesendet:* Donnerstag, 24. März 2016 22:16 >>>>> *An:* mkgmap-dev at lists.mkgmap.org.uk >>>>> *Betreff:* Re: [mkgmap-dev] Problem with splitter: Failed to find >>>>> a correct split >>>>> Gerd, >>>>> >>>>> Sorry, your explanations came in during I was writing up the test >>>>> results ... >>>>> >>>>> Think it's all clear so far. >>>>> >>>>> As we're creating lot of different maps I'm just wondering if I >>>>> can/should drop the option --no-trim for all map building or if I >>>>> would suddenly run into other/new problems... >>>>> >>>>> I'll contact geofabrik with the details. >>>>> >>>>> Many thanks and happy Easter. >>>>> Patrik >>>>> >>>>> On 24.03.2016 22:07, Gerd Petermann wrote: >>>>>> >>>>>> Hi Patrik, >>>>>> >>>>>> >>>>>> I don't need the files, I downloaded the alaska file and tried >>>>>> some variants. >>>>>> >>>>>> See also my last post, send a few minutes ago. >>>>>> >>>>>> >>>>>> Gerd >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im >>>>>> Auftrag von Patrik Brunner <patrik.brunner at gmx.net> >>>>>> *Gesendet:* Donnerstag, 24. März 2016 22:03 >>>>>> *An:* mkgmap-dev at lists.mkgmap.org.uk >>>>>> *Betreff:* Re: [mkgmap-dev] Problem with splitter: Failed to find >>>>>> a correct split >>>>>> Gerd, >>>>>> >>>>>> Yes, alaska.osm.pbf is small. >>>>>> >>>>>> It works without --no-trim. And it also works with the polygon >>>>>> file that belongs to alaska.osm.pbf (also downloaded from >>>>>> Geofabrik) which, according to documentation, anyway would >>>>>> disable --no-trim option. >>>>>> >>>>>> Do you still need the resulting densities-out of the failure ? >>>>>> It's about 700 kb... if yes, how should I provide it ? Just >>>>>> attach it here ? >>>>>> >>>>>> You have to excuse my question, I'm not the crack: is this now a >>>>>> problem of the pbf file, or the splitter ? ... or just the way I >>>>>> try to use it ? >>>>>> >>>>>> Thanks already now for your help. >>>>>> Patrik >>>>>> >>>>>> On 24.03.2016 21:14, Gerd Petermann wrote: >>>>>>> >>>>>>> >>>>>>> Ahh, sorry, I just noticed that the file alaska.osm.pbf is small. >>>>>>> >>>>>>> The problem is here is that the bounding box spans from -180 to >>>>>>> 180, >>>>>>> >>>>>>> but this box is most empty. I have to run splitter now to find >>>>>>> the details. >>>>>>> >>>>>>> It works without --no-trim, probably also with an appropriate >>>>>>> polygon file. >>>>>>> >>>>>>> Does that help? >>>>>>> >>>>>>> >>>>>>> Gerd >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im >>>>>>> Auftrag von Gerd Petermann <GPetermann_muenchen at hotmail.com> >>>>>>> *Gesendet:* Donnerstag, 24. März 2016 21:01 >>>>>>> *An:* Development list for mkgmap >>>>>>> *Betreff:* Re: [mkgmap-dev] Problem with splitter: Failed to >>>>>>> find a correct split >>>>>>> >>>>>>> Hi Patrik, >>>>>>> >>>>>>> >>>>>>> please provide the complete log from splitter and the >>>>>>> densities-out.txt >>>>>>> >>>>>>> Gerd >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> *Von:* mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im >>>>>>> Auftrag von KeenOnKites <keenonkites at gmx.net> >>>>>>> *Gesendet:* Donnerstag, 24. März 2016 20:25 >>>>>>> *An:* Development list for mkgmap >>>>>>> *Betreff:* [mkgmap-dev] Problem with splitter: Failed to find a >>>>>>> correct split >>>>>>> Hello together, >>>>>>> >>>>>>> I'm running into a problem with the splitter (r435 aswell as >>>>>>> r427) when splitting the US_ALASKA file downloadable from Geofabrik. >>>>>>> >>>>>>> The exception is: >>>>>>> >>>>>>> Warning: No solution found for partition >>>>>>> (49.7900390625,-179.9560546875) to (73.828125,180.0) with >>>>>>> 6'702'717 nodes >>>>>>> uk.me.parabola.splitter.SplitFailedException: Failed to find >>>>>>> a correct split >>>>>>> at >>>>>>> uk.me.parabola.splitter.SplittableDensityArea.split(SplittableDensityArea.java:152) >>>>>>> at >>>>>>> uk.me.parabola.splitter.SplittableDensityArea.split(SplittableDensityArea.java:196) >>>>>>> at >>>>>>> uk.me.parabola.splitter.Main.calculateAreas(Main.java:645) >>>>>>> at uk.me.parabola.splitter.Main.split(Main.java:258) >>>>>>> at uk.me.parabola.splitter.Main.start(Main.java:187) >>>>>>> at uk.me.parabola.splitter.Main.main(Main.java:157) >>>>>>> >>>>>>> >>>>>>> The complete command line with the splitter call is: >>>>>>> >>>>>>> java -Xmx1536M -jar >>>>>>> D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/tools/splitter/splitter.jar >>>>>>> --max-threads=2 >>>>>>> --geonames-file=D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/cities/c >>>>>>> ities15000.zip --no-trim >>>>>>> --precomp-sea=D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/sea >>>>>>> --keep-complete=true --mapid=98200001 --max-nodes=800000 >>>>>>> --output=xml --output-dir=D:/fzk/develop/fzk >>>>>>> -mde-garmin/Freizeitkarte-Entwicklung/work/Freizeitkarte_US_ALASKA >>>>>>> D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/work/Freizeitkarte_US_ALASKA/Freizeitkarte_US_ALASKA.osm.pbf >>>>>>> >>>>>>> >>>>>>> The pbf source file comes from: >>>>>>> >>>>>>> http://download.geofabrik.de/north-america/us/alaska-latest.osm.pbf >>>>>>> >>>>>>> >>>>>>> The osmconvert statistics about that file is: >>>>>>> >>>>>>> PS Freizeitkarte-Entwicklung> >>>>>>> .\tools\osmconvert\windows\osmconvert.exe >>>>>>> .\work\Freizeitkarte_US_ALASKA\Freizeitkarte_US_ALASKA.osm.pbf >>>>>>> --out-statistics >>>>>>> timestamp min: 2007-06-05T03:23:59Z >>>>>>> timestamp max: 2016-03-23T05:41:43Z >>>>>>> lon min: -180.0000000 >>>>>>> lon max: -122.5122525 >>>>>>> lat min: 48.6234931 >>>>>>> lat max: 71.6061501 >>>>>>> nodes: 4360214 >>>>>>> ways: 185550 >>>>>>> relations: 2245 >>>>>>> node id min: 27207079 >>>>>>> node id max: 4072166815 >>>>>>> way id min: 4708608 >>>>>>> way id max: 404980503 >>>>>>> relation id min: 13971 >>>>>>> relation id max: 6033189 >>>>>>> keyval pairs max: 310 >>>>>>> keyval pairs max object: relation 60189 >>>>>>> noderefs max: 2000 >>>>>>> noderefs max object: way 42394334 >>>>>>> relrefs max: 681 >>>>>>> relrefs max object: relation 3337277 >>>>>>> PS Freizeitkarte-Entwicklung> >>>>>>> >>>>>>> Interesting to mention: >>>>>>> - splitter exception mentiones a complete different coverage >>>>>>> area than osmconvert statistics. >>>>>>> - the area is near -180 / +180... always complicated. >>>>>>> >>>>>>> Do I miss something ? All other pbf's I've tried are splitting >>>>>>> properly without any problems. Do I need to change something in >>>>>>> the arguments ? Or is it a simple problem of the actual pbf file ? >>>>>>> >>>>>>> Any ideas are very welcome.... >>>>>>> >>>>>>> Cheers >>>>>>> Patrik >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mkgmap-dev mailing list >>>>>>> mkgmap-dev at lists.mkgmap.org.uk >>>>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mkgmap-dev mailing list >>>>>> mkgmap-dev at lists.mkgmap.org.uk >>>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mkgmap-dev mailing list >>>>> mkgmap-dev at lists.mkgmap.org.uk >>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>> >>>> >>>> >>>> _______________________________________________ >>>> mkgmap-dev mailing list >>>> mkgmap-dev at lists.mkgmap.org.uk >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> >>> >>> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev at lists.mkgmap.org.uk >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev at lists.mkgmap.org.uk >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20160325/d03f59ab/attachment-0001.html>
- Previous message: [mkgmap-dev] Problem with splitter: Failed to find a correct split
- Next message: [mkgmap-dev] Problem with splitter: Failed to find a correct split
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list