[mkgmap-dev] Errors when using --gmapi-minimal
From dfjkman at gmail.com dfjkman at gmail.com on Mon May 30 12:12:35 BST 2022
Hi Gerd, I have been away from my computer for a while hence the delayed response. Thanks for your patience on this. I think I understand now but it would help with clarity if you could provide me with the commands you would use in my situation. For example I would normally download Zambia-latest.osm.pbf and then use splitter to split the file into smaller sections. The resulting template.args would then be used with -c to provide the input files for mkgmap which creates the .img files that now make up the map. In my case I would like to use the above procedure to take the latest OSM data and combine it with DEM data. I could provide this DEM input by using -c and the template.args file created when I split the original DEM data but that would mean recreating .img files I already have. I therefore want to reuse these .img files that are now in C:\Users\Dave\Documents\Maps\Data\DemImg\. My current understanding is that if I provide the input files for the new OSM data using -c C:\Users\Dave\Documents\Maps\Temp\Split_files\template.args and then provide the DEM data input using C:\Users\Dave\Documents\Maps\Data\DemImg\3641*.img with --gmapi-minimal I should get a .gmap folder containing map tiles of both the new data and the old existing DEM data. What I have found is that if I do not provide an include-pattern with the --gmapi-minimal command the new map does not contain the information from the DEM .img files from C:\Users\Dave\Documents\Maps\Data\DemImg\, they are skipped. However there are empty folders created in the Product1 folder within the .gmapi folder, one for each .img file in the DEM input folder covered by the '3641*.img' mask i.e. there are empty folders named 3641, 3642 and so on. This would imply that the include-pattern is required rather than optional in my situation. It also explains the 'missing maps' errors flagged by JaVaWa. By extension it would also explain what is happening when the include-pattern does not include all the files within the input folder, again empty folders are being created for each skipped .img file. Dave -----Original Message----- From: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> On Behalf Of Gerd Petermann Sent: 27 May 2022 09:34 To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Dave, I agree that the --gmapi-minimal option is quite special. The code to implement the optional pattern stores the pattern in a variable called mustWritePattern. Then, mkgmap first determines that a file should be written because it is in the list of input files, then - because of the gmapi-minimal option - it detects that the output file already exists and would skip it but the mustWritePattern says that it should still be overwritten. Hope this makes it clearer? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von dfjkman at gmail.com <dfjkman at gmail.com> Gesendet: Freitag, 27. Mai 2022 07:55 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Gerd, In my situation the problem is solved, your understanding of my case is correct. I always knew that if I removed the overview files from the input folder I would have no problem, your suggestion of a better mask means I do not have to remove these files, thanks for that. I was just thinking that if, for example, you had an input folder containing contour files for a large area and only wanted to create a map of a much smaller area the include pattern was a way to do this. This in fact works but does mean that the files that are in the input folder but are skipped in the process are still referenced in the tdb. I suspect, but am no expert obviously, the content of the tdb is governed by the initial input instruction and mask, then later the --gmapi-minimal pattern skips some files that have been already referenced in the tdb. While this may be something that is never used it is something that can be done and the resultant missing references are not fatal and do not cause BaseCamp to shut down. References to missing overview files however are fatal as I have discovered. Thanks for all your work on mkgmap, the maps I produce using it help me to hunt down the many mapping errors made by remote mappers and the many inexperienced mappers mapping in Zambia. Kind regards, Dave -----Original Message----- From: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> On Behalf Of Gerd Petermann Sent: 27 May 2022 07:19 To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Dave, does that mean the problem is completely solved now? My current understanding is that you created the gmapi folder, later created gmapsupp and overmap files in the folder with the DEM tiles and that those additional files were selected as input files with the *.img mask. The pattern for the --gmapi-minimal option really just forces a copying of files, it has no influence on the tdb content. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von dfjkman at gmail.com <dfjkman at gmail.com> Gesendet: Donnerstag, 26. Mai 2022 23:24 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Gerd, Having a more specific mask as you suggest solves the problem of any overview map or gmapsupp being included as input, but it does not get round the fact that any files excluded by --gmapi-minimal, with include-pattern specified, are still included in the TDB file but not present in the map. It does mean that BaseCamp does not crash as any missing files are not overview ones. It does not really affect me as I will not be filtering my input files it is just that I noticed this behaviour when using the include-pattern with --gmapi-minimal, using the improved mask as you suggested will exclude any overview map anyway in my situation, I have just checked. When I used --gmapi to combine the maps I did not have the overview or gmapsupp files in my input folders, only the files I required, so did not come across the problem. It was only after I had recreated the DEM and contour maps that I came across the problem as new overview maps and gmapsupp had been created and I had not removed them. I then had a look at using the include-pattern with --gmapi-minimal to prevent the files being used as input and so started to play around with various patterns to see the effects. I just thought I would let you know my findings. Perhaps I misunderstood the purpose of the include-pattern. Dave -----Original Message----- From: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> On Behalf Of Gerd Petermann Sent: 26 May 2022 20:35 To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Dave, so I think your problem is not the --gmapi-minimal option. You probably get the same problems with --gmapi. Make sure that you don't include files which are not wanted in the target directory. So, instead of C:\Users\Dave\Documents\Maps\Data\DemImg\*.img use a more specifc mask like C:\Users\Dave\Documents\Maps\Data\DemImg\3641*.img so that only the wanted tiles and not the overview map or a gmapsupp is included. Once this works well with the --gmapi option you can use the --gmapi-minimal option to avoid the redundant copying of those tiles. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von dfjkman at gmail.com <dfjkman at gmail.com> Gesendet: Donnerstag, 26. Mai 2022 19:05 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Gerd, I have previously created a separate DEM map containing only DEM data and all .img files are stored in a folder called ' C:\Users\Dave\Documents\Maps\Data\DemImg\' as well as a separate map containing only contours with all .img files stored in a folder called ' C:\Users\Dave\Documents\Maps\Data\ContImg\', both are transparent and I had used the individual renamed gmapsupp.img maps on my Etrex to be able to show DEM or contour maps together with my normal map of Zambia by enabling or disabling them on the device. I tried my normal map building procedure using the --gmapi-minimal option to combine the existing DEM and Contour maps with my standard map for use in BaseCamp at first just as an experiment and did not use any filters and had no problems as the DemImg and ContImg folders only had the required .img files, no overview map. I then Recreated the DEM and contour maps changing the DEM distances and contour intervals and then recreated my combined map which is when I noticed the error message. On investigating I realized the 2 folders now contained the overview maps, ' C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img' and C:\Users\Dave\Documents\Maps\Data\ContImg\Zambia_Contours.img'. Thinking it was because I had not filtered the --gmapi-minimal I used the following filter, .*[0-9]{8}\.img, to select only .img files with names made up of numbers so it would ignore 'Zambia_Contours.img' and 'Zambia_Dem.img'. This it did but there was still the error message from mkgmap and JaVaWa GMTK reported this error ' TDB error: One or more map segments in the TDB file are missing from disk.' and listed the following: 63240000: Overview Map (Zambia_C) (missing) 63240000: Overview Map (Zambia_D) (missing) As a trial I then created a filter, ' .*[0-8]{8}\.img ' to skip .img files containing a 9, so this filter would skip 36410009.img, 36410019.img and so on. Mkgmap reports it is skipping these files as expected but once again JaVaWa GMTK reports: Name on computer: Zam_OSM_Dem_26-05-2022 Name on device: Zam_OSM_Dem_26-05-2022 Family ID (FID): 2636 Product ID (PID): 1 Version: 1.00 Size: 428.62 MiB Location: C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\ TYP file: C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\dfj.typ Map type: .gmap map (global profile) Protected: No TDB error: One or more map segments in the TDB file are missing from disk. Errors recoverable: Maybe Map segments (94): 26350001: ZM-Copperbelt Province (26350001) 26350002: ZM-Western Province (26350002) 26350003: ZM-Chibombo District (26350003) 26350004: ZM-Southern Province (26350004) 26350005: ZM-Mazabuka (26350005) 26350006: ZM-Mongu (26350006) 26350007: ZM-Livingstone (26350007) 26350008: ZM-Northern Province (26350008) 26350009: ZM-Kitwe (26350009) 26350010: ZM-Mpika (26350010) 26350011: ZM-Muchinga Province (26350011) 26350012: ZM-Mkushi (26350012) 26350013: ZM-Eastern Province (26350013) 26350014: ZM-Central Province (26350014) 26350015: ZM-Lusaka Province (26350015) 26350016: ZM-Republic of Zambia (26350016) 36410001: Zambia_Dem (36410001) 36410002: Zambia_Dem (36410002) 36410003: Zambia_Dem (36410003) 36410004: Zambia_Dem (36410004) 36410005: Zambia_Dem (36410005) 36410006: Zambia_Dem (36410006) 36410007: Zambia_Dem (36410007) 36410008: Zambia_Dem (36410008) 36410009: Zambia_Dem (36410009) (missing) 36410010: Zambia_Dem (36410010) 36410011: Zambia_Dem (36410011) 36410012: Zambia_Dem (36410012) 36410013: Zambia_Dem (36410013) 36410014: Zambia_Dem (36410014) 36410015: Zambia_Dem (36410015) 36410016: Zambia_Dem (36410016) 36410017: Zambia_Dem (36410017) 36410018: Zambia_Dem (36410018) 36410019: Zambia_Dem (36410019) (missing) 36410020: Zambia_Dem (36410020) 36410021: Zambia_Dem (36410021) 36410022: Zambia_Dem (36410022) 36410023: Zambia_Dem (36410023) 36410024: Zambia_Dem (36410024) 36410025: Zambia_Dem (36410025) 36410026: Zambia_Dem (36410026) 36410027: Zambia_Dem (36410027) 36410028: Zambia_Dem (36410028) 36410029: Zambia_Dem (36410029) (missing) 36410030: Zambia_Dem (36410030) 36410031: Zambia_Dem (36410031) 36410032: Zambia_Dem (36410032) 36410033: Zambia_Dem (36410033) 36410034: Zambia_Dem (36410034) 36410035: Zambia_Dem (36410035) 36410036: Zambia_Dem (36410036) 36410037: Zambia_Dem (36410037) 36410038: Zambia_Dem (36410038) 36410039: Zambia_Dem (36410039) (missing) 86410001: Zambia_Contours (86410001) 86410002: Zambia_Contours (86410002) 86410003: Zambia_Contours (86410003) 86410004: Zambia_Contours (86410004) 86410005: Zambia_Contours (86410005) 86410006: Zambia_Contours (86410006) 86410007: Zambia_Contours (86410007) 86410008: Zambia_Contours (86410008) 86410009: Zambia_Contours (86410009) (missing) 86410010: Zambia_Contours (86410010) 86410011: Zambia_Contours (86410011) 86410012: Zambia_Contours (86410012) 86410013: Zambia_Contours (86410013) 86410014: Zambia_Contours (86410014) 86410015: Zambia_Contours (86410015) 86410016: Zambia_Contours (86410016) 86410017: Zambia_Contours (86410017) 86410018: Zambia_Contours (86410018) 86410019: Zambia_Contours (86410019) (missing) 86410020: Zambia_Contours (86410020) 86410021: Zambia_Contours (86410021) 86410022: Zambia_Contours (86410022) 86410023: Zambia_Contours (86410023) 86410024: Zambia_Contours (86410024) 86410025: Zambia_Contours (86410025) 86410026: Zambia_Contours (86410026) 86410027: Zambia_Contours (86410027) 86410028: Zambia_Contours (86410028) 86410029: Zambia_Contours (86410029) (missing) 86410030: Zambia_Contours (86410030) 86410031: Zambia_Contours (86410031) 86410032: Zambia_Contours (86410032) 86410033: Zambia_Contours (86410033) 86410034: Zambia_Contours (86410034) 86410035: Zambia_Contours (86410035) 86410036: Zambia_Contours (86410036) 86410037: Zambia_Contours (86410037) 86410038: Zambia_Contours (86410038) 86410039: Zambia_Contours (86410039) (missing) Other essential C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\Product1\Zambia.tdb files/folders: C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\Product1\Zambia C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\Zambia.mdx C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\Zambia_mdr C:\ProgramData\Garmin\Maps\Zam_OSM_Dem_26-05-2022.gmap\info.xml. JaVaWa can repair these errors by rewriting the TDB file to exclude the missing files. I had removed any overview files (Zambia_Contours.img and Zambia_Dem.img) from the input folders, if they are left in they are skipped and mkgmap will report: gmapi-minimal: Skipping file C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img gmapi-minimal: Skipping file C:\Users\Dave\Documents\Maps\Data\ContImg\Zambia_Contours.img WARNING (global): Could not copy 63240000.RGN uk.me.parabola.imgfmt.FileExistsException: File 63240000.RGN already exists WARNING (global): Could not copy 63240000.TRE uk.me.parabola.imgfmt.FileExistsException: File 63240000.TRE already exists WARNING (global): Could not copy 63240000.LBL uk.me.parabola.imgfmt.FileExistsException: File 63240000.LBL already exists SEVERE (global): Invalid mapname for C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img, perhaps it is not a .img file SEVERE (global): Invalid mapname for C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img, perhaps it is not a .img file As you can see from the attached screengrab from BaseCamp of the trial map it behaves as you would expect with the DEM and contour data missing in the area of the skipped file, BaseCamp will load Ok even if the TDB file is not corrected by JaVaWa but it will not load without correction if any of the skipped files are overview files. My normal instruction to create the combined map is as follows: Java -jar -Xmx7g ` C:\Users\Dave\Documents\Maps\Utils\mkgmap\mkgmap.jar ` --style-file=C:\Users\Dave\Documents\Maps\Styles\dfj ` -c C:\Users\Dave\Documents\Maps\Options\Zambia.cfg ` --output-dir= "C:\Users\Dave\Documents\Maps\Temp\Output Map" ` --family-id= 2635 ` --family-name=" Zam_OSM_Dem_26-05-2022" ` --series-name=" Zam_OSM_Dem_26-05-2022" ` --gmapi-minimal='.*[0-9]{8}\.img' ` -c C:\Users\Dave\Documents\Maps\Temp\Split_files\template.args ` <----- new map input C:\Users\Dave\Documents\Maps\Data\DemImg\*.img ` <----- pre compiled DEM map input C:\Users\Dave\Documents\Maps\Data\ContImg\*.img ` <----- pre compiled contour map input -n 26350001 ` --description=Zambia ` C:\Users\Dave\Documents\Maps\Typ\dfj.txt ` The commands to create the initial DEM maps and contour maps are the same as above except not using the --gmapi-minimal command and using C:\Users\Dave\Documents\Maps\Data\DemImg as the output directory for the DEM maps and C:\Users\Dave\Documents\Maps\Data\ContImg for the contour maps and the input files obviously provide the elevation data and nothing else. I hope this is clear. Kind regards, Dave -----Original Message----- From: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> On Behalf Of Gerd Petermann Sent: 26 May 2022 14:27 To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] Errors when using --gmapi-minimal Hi Dave, I don't know if it's an error in mkgmap or maybe a wrong expectation. Please tell us the complete command(s) to create the initial map and the 2. command to uppdate it with --gmapi-minimal. What do you expect to happen with the --gmapi-minimal option? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von dfjkman at gmail.com <dfjkman at gmail.com> Gesendet: Donnerstag, 26. Mai 2022 12:36 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] Errors when using --gmapi-minimal Hi Gerd, I decided to have a go using the --gmapi-minimal option and may have come across a potential bug. When using the include-pattern and some .img files are skipped because they do not match the pattern, but exist in the input folder containing pre written files, these skipped files are still written to the *.tdb file but not included in the map. In other words using the pattern example given --gmapi-minimal=.*4711[0-9]{4}\.img, matching files between 47110000.img and 47119999.img, if there is also a file 47120000.img this will be skipped but there will still be a reference to the skipped in the *.tdb file the same with any overview files that may exist in the input folder. I think having any overview maps in the input folder will generate the following error messages: gmapi-minimal: Skipping file C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img gmapi-minimal: Skipping file C:\Users\Dave\Documents\Maps\Data\ContImg\Zambia_Contours.img WARNING (global): Could not copy 63240000.RGN uk.me.parabola.imgfmt.FileExistsException: File 63240000.RGN already exists WARNING (global): Could not copy 63240000.TRE uk.me.parabola.imgfmt.FileExistsException: File 63240000.TRE already exists WARNING (global): Could not copy 63240000.LBL uk.me.parabola.imgfmt.FileExistsException: File 63240000.LBL already exists SEVERE (global): Invalid mapname for C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img, perhaps it is not a .img file SEVERE (global): Invalid mapname for C:\Users\Dave\Documents\Maps\Data\DemImg\Zambia_Dem.img, perhaps it is not a .img file I discovered the errors while using JaVaWa GMTK when checking the maps for errors. I am creating a map including pre written DEM and contour information to try find badly mapped rivers that go uphill and over ridges in BaseCamp. Kind regards, Dave _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
- Previous message: [mkgmap-dev] Errors when using --gmapi-minimal
- Next message: [mkgmap-dev] Errors when using --gmapi-minimal
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list