[mkgmap-dev] splitter exception with SRTM data for USA
From Gerd Petermann gpetermann_muenchen at hotmail.com on Thu Apr 11 18:18:34 BST 2013
Hi Thorsten, good to know. Yes, the smaller the tiles, the less likely you hit the case were the limit is involved. It happens when many consecutive nodes lie in the same area, this gives a special pattern which is saved in one data structure that can only hold up to 2^31 items. I have to do some tests regarding the performance with the old code. I think it was ~ 10% lower memory usage, so I think this is not a good reason to use code that has known limitations and is a lot more complex. Gerd > Date: Thu, 11 Apr 2013 18:51:34 +0200 > From: kukuk at suse.de > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA > > > Hi, > > On Thu, Apr 11, Gerd Petermann wrote: > > > Hi Thorsten, > > > > I fear my idea regarding the numering of node ids will not help. > > The problem is the amount of nodes and the fact that the generated data probably > > is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close > > together. This makes it very likely that splitter reaches a limit of 2.147.483.648. > > With normal OSM data, this is very unlikely. > > > > I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit. > > It looks like splitter will work if I lower the limit of max. nodes > in a tile. > I don't know which improvements your optimizations are giving, but > I don't think that we need to care for this very special case, especially > as it looks like I found a workaround. But with this big amount of data, > testing will need some more time. > > Thorsten > > > > Gerd > > > > > Date: Thu, 11 Apr 2013 16:54:13 +0200 > > > From: kukuk at suse.de > > > To: mkgmap-dev at lists.mkgmap.org.uk > > > Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA > > > > > > > > > Hi Gerd, > > > > > > On Thu, Apr 11, GerdP wrote: > > > > > > > Hi Thorsten, > > > > > > > > ok, so I assume that you can patch pyghtmap to generate the ids as I > > > > requested. > > > > > > Yes, I will try that over the weekend, generating such big data needs > > > some time. > > > > > > Thorsten > > > > > > > In the mean time I will write a small generator which creates OSM data with > > > > a large number of nodes. Maybe it is a bug in splitter, not a limit. > > > > > > > > Gerd > > > > > > > > > > > > Thorsten Kukuk wrote > > > > > Hi, > > > > > > > > > > On Thu, Apr 11, Felix Hartmann wrote: > > > > > > > > > >> are you using the newest version of pyghtmap, newest splitter, and no > > > > >> older than 3-4 weeks mkgmap? > > > > >> > > > > >> I had a similar problem with Asia. mkgmap splitter wouldn't split the > > > > >> pyghtmap extract - no matter what I tried (splitter didn't split but > > > > >> simply output one huge file instead). (about 6 month ago) - 1 month ago > > > > >> I redid everything using newest versions, and it worked... > > > > > > > > > > Since I wrote a lot of the patches for the last phyghtmap versions, > > > > > I use of course the newest one. > > > > > > > > > > And it is only with the US. Canada, but that's a little bit > > > > > smaller, doesn't create any problems. > > > > > > > > > > Thorsten > > > > > > > > > >> On 11.04.2013 16:33, Thorsten Kukuk wrote: > > > > >> > Hi Gerd, > > > > >> > > > > > >> > On Thu, Apr 11, Gerd Petermann wrote: > > > > >> > > > > > >> >> Hi Thorsten, > > > > >> >> > > > > >> >> another question: The wiki of phygthmap says that you don't need > > > > >> splitting: > > > > >> >> http://wiki.openstreetmap.org/wiki/Phyghtmap > > > > >> >> > > > > >> >> Why do you do it? > > > > >> > Because I create one big data file with phyghtmap and then extract > > > > >> > the parts I need. That's much easier and simpler then to let phyghtmap > > > > >> > create a big amount of files and try to find and cut the ones you need. > > > > >> > > > > > >> > Thorsten > > > > >> > > > > > >> >> Gerd > > > > >> >> > > > > >> >>> Date: Thu, 11 Apr 2013 14:06:30 +0200 > > > > >> >>> From: > > > > > > > > > kukuk@ > > > > > > > > >> >>> To: > > > > > > > > > mkgmap-dev at .org > > > > > > > > >> >>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA > > > > >> >>> > > > > >> >>> > > > > >> >>> Hi, > > > > >> >>> > > > > >> >>> I generated SRTM data for the USA with phygthmap and wanted to try > > > > >> >>> to split it now for compiling with mkgmap. But while this worked fine > > > > >> >>> for all other countries, in this case I get always the following > > > > >> >>> exception: > > > > >> >>> > > > > >> >>> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 > > > > >> of 65535 > > > > >> >>> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead > > > > >> includes 19 arrays with 8 MB > > > > >> >>> 2.350.000.000 nodes processed... id=2526831732 > > > > >> >>> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 > > > > >> of 65535 > > > > >> >>> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead > > > > >> includes 19 arrays with 8 MB > > > > >> >>> 2.360.000.000 nodes processed... id=2536831732 > > > > >> >>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: > > > > >> 524288 > > > > >> >>> at > > > > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) > > > > >> >>> at crosby.binary.BinaryParser.parse(Unknown Source) > > > > >> >>> at crosby.binary.BinaryParser.handleBlock(Unknown Source) > > > > >> >>> at crosby.binary.file.FileBlock.process(Unknown Source) > > > > >> >>> at crosby.binary.file.BlockInputStream.process(Unknown > > > > >> Source) > > > > >> >>> at uk.me.parabola.splitter.Main.processMap(Main.java:792) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) > > > > >> >>> at > > > > >> uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) > > > > >> >>> at uk.me.parabola.splitter.Main.split(Main.java:244) > > > > >> >>> at uk.me.parabola.splitter.Main.start(Main.java:157) > > > > >> >>> at uk.me.parabola.splitter.Main.main(Main.java:146) > > > > >> >>> > > > > >> >>> Any ideas? It's splitter r300. > > > > >> >>> > > > > >> >>> Splitter version unknown compiled 2013-04-10T21:31:05+0000 > > > > >> >>> boundary-tags=use-exclude-list > > > > >> >>> cache= > > > > >> >>> description=TK-USA-Tile > > > > >> >>> geonames-file=osmmaps/scripts/cities/USA.txt > > > > >> >>> keep-complete=true > > > > >> >>> mapid=71510001 > > > > >> >>> max-areas=1024 > > > > >> >>> max-nodes=5000000 > > > > >> >>> max-threads=4 (auto) > > > > >> >>> mixed=false > > > > >> >>> no-trim=false > > > > >> >>> output=pbf > > > > >> >>> output-dir=build/usa/tiles-srtm > > > > >> >>> overlap=0 > > > > >> >>> polygon-file= > > > > >> >>> precomp-sea=data/sea/current > > > > >> >>> problem-file= > > > > >> >>> problem-report= > > > > >> >>> resolution=13 > > > > >> >>> split-file= > > > > >> >>> status-freq=120 > > > > >> >>> stop-after=dist > > > > >> >>> write-kml= > > > > >> >>> > > > > >> >>> > > > > >> >>> -- > > > > >> >>> Thorsten Kukuk, Project Manager/Release Manager SLES > > > > >> >>> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg > > > > >> >>> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG > > > > >> Nürnberg) > > > > >> >>> _______________________________________________ > > > > >> >>> mkgmap-dev mailing list > > > > >> >>> > > > > > > > > > mkgmap-dev at .org > > > > > > > > >> >>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > >> >> > > > > >> >> _______________________________________________ > > > > >> >> mkgmap-dev mailing list > > > > >> >> > > > > > > > > > mkgmap-dev at .org > > > > > > > > >> >> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > >> > > > > >> -- > > > > >> keep on biking and discovering new trails > > > > >> > > > > >> Felix > > > > >> openmtbmap.org & www.velomap.org > > > > >> > > > > >> > > > > >> _______________________________________________ > > > > >> mkgmap-dev mailing list > > > > >> > > > > > > > > > mkgmap-dev at .org > > > > > > > > >> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > > > > > -- > > > > > Thorsten Kukuk, Project Manager/Release Manager SLES > > > > > SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg > > > > > GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) > > > > > _______________________________________________ > > > > > mkgmap-dev mailing list > > > > > > > > > mkgmap-dev at .org > > > > > > > > > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5756629p5756654.html > > > > Sent from the Mkgmap Development mailing list archive at Nabble.com. > > > > _______________________________________________ > > > > mkgmap-dev mailing list > > > > mkgmap-dev at lists.mkgmap.org.uk > > > > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > > > Thorsten Kukuk, Project Manager/Release Manager SLES > > > SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg > > > GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) > > > _______________________________________________ > > > mkgmap-dev mailing list > > > mkgmap-dev at lists.mkgmap.org.uk > > > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk > > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > -- > Thorsten Kukuk, Project Manager/Release Manager SLES > SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg > GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20130411/1a2f3fbe/attachment-0001.html
- Previous message: [mkgmap-dev] splitter exception with SRTM data for USA
- Next message: [mkgmap-dev] splitter exception with SRTM data for USA
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list