[mkgmap-dev] [PATCH v2] Performance improvement by removing unused elements before the style processing

Wed Jan 4 20:14:06 GMT 2012

Steve,

can you do me a favour?
I have uploaded both versions to http://files.mkgmap.org.uk/detail/46 
(only files that differ and not the gmapsupp.img).

Can you run the display tool and check if you can quickly find the 
difference? I would have to completely install the display tool and have 
to get warm with it... I think you are much quicker? ;-)

Thanks!
WanMil

> Hello WanMil,
>
> I can reproduce the problem with the attached files like this:
> 1) download saarland.osm.pbf from geofabrik created 04-Jan-2012 04:36
> 2) Use splitter r198 with --max-nodes=200000
> 3) compile r2160 with my identical_output.patch
> 4) execute mkgmap with the parms in test.bat
> 5) copy the output files to somewhwere else
> 6) execute mkgmap again with the same commands, verify that the new
> output files are identical to the copy
> 7) compile r2160 with my identical_output.patch +
> c:\TEMP\remove_unused_elements_v2.patch
> 8) execute mkgmap again with the same commands. I would again expect
> identical output, but I see this
> C:\temp\prove_patch>diff -qb ..\prove .
> Files ..\prove\63240003.img and .\63240003.img differ
> Files ..\prove\gmapsupp.img and .\gmapsupp.img differ
> Files ..\prove\mkgmap.jar and .\mkgmap.jar differ
> Files ..\prove\osmmap.tdb and .\osmmap.tdb differ
> Files ..\prove\osmmap_mdr.img and .\osmmap_mdr.img differ
>
> Ciao,
> Gerd
>
>
>
>
>  > Date: Wed, 4 Jan 2012 19:10:28 +0100
>  > From: wmgcnfg at web.de
>  > To: mkgmap-dev at lists.mkgmap.org.uk
>  > Subject: Re: [mkgmap-dev] [PATCH v2] Performance improvement by
> removing unused elements before the style processing
>  >
>  > Hi Gerd,
>  >
>  > yes of course I am interested, Send me all information you have.
>  >
>  > Thanks
>  > WanMil
>  >
>  > > Hi WanMil,
>  > >
>  > > I tested the patch with my tiles for Saarland. I can confirm a
> reduction of
>  > > processing time ~ 6% compared to r2160, and also a reduction for
> the peek
>  > > value of heap memory usage (238M -> 210M), so that's both good!
>  > > BUT I also see a difference in one of the seven output img files
> (plus the
>  > > resulting gmapsupp.img), and I think this is not intended. Both
> files have
>  > > the same size, but are different in many bytes.
>  > > Let me know when I should send details, maybe the new result is
> better than
>  > > the old ;-)
>  > >
>  > > Ciao,
>  > > Gerd
>  > >
>  > >
>  > > WanMil wrote
>  > >>
>  > >> The 2nd patch fixes the remaining problems:
>  > >> 1. Ways without tags but referenced by relations are no longer removed
>  > >> (they might be tagged by the relation style file and could therefore
>  > >> appear in the map)
>  > >> 2. Intersection of ways with the tile bounding box is now checked
>  > >> instead of only checking that at least one point is contained in the
>  > >> bounding box. Only very few ways are affected by this but otherwise
>  > >> routing problems are possible.
>  > >>
>  > >> The performance improvement seems to be good (measured with my
> test map):
>  > >> r2159: ~250s
>  > >> patched: ~225s
>  > >>
>  > >> I also expect (although I haven't tested) that the max memory
>  > >> requirement of mkgmap is decreased. I think max mem is used when the
>  > >> style file is just processed. At this stage all raw OSM elements
> and all
>  > >> style file processed elements are kept in memory. With the patch the
>  > >> number of OSM elements is noticeably reduced.
>  > >>
>  > >> WanMil
>  > >>
>  > >>> Hi,
>  > >>>
>  > >>> this is another performance improvement:
>  > >>>
>  > >>> Usually the mkgmap input tiles are larger than the processed bounding
>  > >>> box (splitter parameter overlap). So there are much many elements
> which
>  > >>> are processed but thrown away at a late step in mkgmap.
>  > >>>
>  > >>> The patch tries to remove them much earlier before the style
> files are
>  > >>> processed and before the LocationHook starts (which ignores them but
>  > >>> that must also be calculated).
>  > >>>
>  > >>> The patch contains one drawback:
>  > >>> Ways which have all its points outside the bounding box of the
> tile but
>  > >>> which cross the tile are also removed. If that's a point the
> patch must
>  > >>> be improved.
>  > >>>
>  > >>> Have fun!
>  > >>> WanMil
>  > >>>
>  > >>>
>  > >>> _______________________________________________
>  > >>> mkgmap-dev mailing list
>  > >>> mkgmap-dev at .org
>  > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>  > >>
>  > >>
>  > >> _______________________________________________
>  > >> mkgmap-dev mailing list
>  > >> mkgmap-dev at .org
>  > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>  > >>
>  > >
>  > >
>  > > --
>  > > View this message in context:
> http://gis.638310.n2.nabble.com/PATCH-v1-Performance-improvement-by-removing-unused-elements-before-the-style-processing-tp7144978p7149735.html
>  > > Sent from the Mkgmap Development mailing list archive at Nabble.com.
>  > > _______________________________________________
>  > > mkgmap-dev mailing list
>  > > mkgmap-dev at lists.mkgmap.org.uk
>  > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>  >
>  > _______________________________________________
>  > mkgmap-dev mailing list
>  > mkgmap-dev at lists.mkgmap.org.uk
>  > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev