[mkgmap-dev] New branch for default typ file

Sat Dec 14 15:44:13 GMT 2019

Dear Sirs:

Re UTF-8

Microsoft does this to handle another problem: Microsoft uses many 
different character set encodings (i.e., code pages, e.g., CP1252) and 
the BOM is used by Microsoft to indicate that the "code page" is UTF-8. 
Java was implemented to an older version of the Unicode standard that 
prohibited an UTF-8 BOM. The problem is comes from moving back and forth 
across that cultural divide. Yes, this is painful.

A solution to the reading issue from the Java side:

https://stackoverflow.com/questions/1835430/byte-order-mark-screws-up-file-reading-in-java

A solution for writing a UTF-8 BOM in Java:

|BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new 
FileOutputStream(the File), StandardCharsets.UTF_8))out.write('\ufeff');|

A check for execution in a Windows environment:

String OS = System.getProperty("os.name").toLowerCase();
Boolean isWindows = OS.indexOf("win") >= 0;

Perhaps, on output write the BOM in a Windows environment and use the 
BOM optional on input.

http://www.unicode.org/faq/utf_bom.html

Q: What are some of the differences between the UTFs?

A: The following table summarizes some of the properties of each of the 
UTFs.

Name 	UTF-8 	UTF-16 	UTF-16BE 	UTF-16LE 	UTF-32 	UTF-32BE 	UTF-32LE
Smallest code point 	0000 	0000 	0000 	0000 	0000 	0000 	0000
Largest code point 	10FFFF 	10FFFF 	10FFFF 	10FFFF 	10FFFF 	10FFFF 	10FFFF
Code unit size 	8 bits 	16 bits 	16 bits 	16 bits 	32 bits 	32 bits 	32 
bits
Byte order 	N/A 	<BOM> 	big-endian 	little-endian 	<BOM> 	big-endian 
little-endian
Fewest bytes per character 	1 	2 	2 	2 	4 	4 	4
Most bytes per character 	4 	4 	4 	4 	4 	4 	4

In the table <BOM> indicates that the byte order is determined by a byte 
order mark, if present at the beginning of the data stream, otherwise it 
is big-endian.

http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

Table 2-4.  The Seven Unicode Encoding Schemes

Encoding Scheme       Endian Order                    BOM Allowed?

UTF-8 N/A                                         yes

The remainder of the table omitted.

https://docs.microsoft.com/en-us/windows/win32/intl/using-byte-order-marks

  Using Byte Order Marks

  * 05/30/2018
  * 2 minutes to read
  *
      o
      o
    <https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/Intl/using-byte-order-marks.md>

Always prefix a Unicode plain text file with a byte order mark, which 
informs an application receiving the file that the file is byte-ordered. 
Available byte order marks are listed in the following table. Because 
Unicode plain text is a sequence of 16-bit code values, it is sensitive 
to the byte ordering used when the text is written.

Note

A byte order mark is not a control character that selects the byte order 
of the text.

Byte order mark 	Description
EF BB BF 	UTF-8
FF FE 	UTF-16, little endian
FE FF 	UTF-16, big endian
FF FE 00 00 	UTF-32, little endian
00 00 FE FF 	UTF-32, big-endian

On 12/14/2019 3:41 AM, Ticker Berkin wrote:
> Hi Joris & Gerd
>
> Great to see the typ-files now in trunk and all the work in updating
> mapnik.txt to the current default style. Next week I plan to go through
> "20191209 mapnik update.pdf" and comment on it and possible changes to
> the default style.
>
> Some other questions however:
>
> How do you see mapnik.txt now being maintained; will it be as as simple
> .txt file with patches being supplied in the same way as other source
> files, or will it be regenerated from your translation spreadsheet and
> other sources? I'd prefer the simple text file approach, but this might
> allow changes into the file which make it incompatible with the tools
> Joris uses to enhance it.
>
> It is currently in UTF8 format, with an appropriate BOM at the start of
> the file. I don't know how the java input libraries determine the
> conversion rules to internal unicode, but this file should be
> consistent with all the others that contain characters outside the
> simple ansi 7-bit range (roadNameConfig.txt, default/inc/address)
>
> It contains the statement:
>> CodePage=65001
> This is saying the output should be unicode, but the output should be
> the same as the associated map.
>
> Also the FID should be removed.
>
> Regards
> Ticker
>
> On Tue, 2019-12-10 at 09:59 +0000, Gerd Petermann wrote:
>> Hi Joris,
>>
>> the file mapnik.txt says "Based on mkgmap default style version:
>> r4262"
>> Is it the right file?
>>
>> reg. line type 0x0b: highway=motorway_link & (mkgmap:exit_hint=true |
>> mkgmap:dest_hint=*)
>> I want to look at the DestinationHook. If I got that right it should
>> be OK to have a zero-length road with that type to get the wanted
>> destination hint. In that case we don't have to care about rendering.
>>
>> Gerd
>>
>> ________________________________________
>> Von: Joris Bo <jorisbo at hotmail.com>
>> Gesendet: Montag, 9. Dezember 2019 20:45
>> An: Development list for mkgmap; Gerd Petermann
>> Betreff: RE: [mkgmap-dev] New branch for default typ file
>>
>> Hi All,
>>
>> I don't think any changes needed in mkgmap itself. When the draworder
>> of bay is lower then water it will display correctly.
>> See attached new typ-file for correct usage.
>> Even better (but this is a change in default style): don't use
>> natural = bay in polygons but only in points for displaying as name.
>>
>> Today I spent some time testing and repairing.
>>
>> The mapnik.txt in branch mkgmap-default-typ-r4268 was pretty old and
>> also did not have the translations of all the languages anymore. It
>> also lost draworder of a lot of polygons which made the bay-problem
>> occur.
>>
>> I did a complete recheck of the most recent default-style in: mkgmap
>> -r4386.zip and changed de typ-file accordingly.
>>
>> I downloaded a full europe-latest from geofabrik today, builded it as
>> a big full europa map with the default style of r4386  and with
>> mkgmap r4386.jar No errors occured.
>>
>> I think it’s up to date again but some review and comments are always
>> welcome.
>>
>> See typ-file in attachement,
>>
>> Kind regards,
>> Joris
>>
>>
>>
>>
>>
>> -----Oorspronkelijk bericht-----
>> Van: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> Namens Pinns
>> UK
>> Verzonden: maandag 9 december 2019 18:31
>> Aan: Gerd Petermann <gpetermann_muenchen at hotmail.com>;
>> mkgmap-dev at lists.mkgmap.org.uk
>> Onderwerp: Re: [mkgmap-dev] New branch for default typ file
>>
>> Hi Gerd
>>
>> Yes, you can do that with a draw level 1 higher than sea.
>>
>> Draw orders are defined at the beginning of a (txt) typ file just
>> before the polygons
>>
>> using the following format
>>
>> Type=0x type number , draworder
>>
>> It is good practice to sort the draworders , as that is how they
>> appear in a typ file
>>
>> [_drawOrder]
>> Type=0x03,1
>> Type=0x28,1
>> Type=0x54,1
>> Type=0x01,2
>> Type=0x09,2
>>    Type=0x4E,2
>>    Type=0x10F1C,2
>> etc etc
>> [end]
>> I have no idea what the draworder for sea is , but just make it one
>> higher
>>
>> On 09/12/2019 16:41, Gerd Petermann wrote:
>>> Hi Nick,
>>>
>>> I don't want to cut out islands from bay polygons, I thought about
>>> a proper typ for 0x3d which somehow marks "calmer water"
>>> and a draw order that puts this above water and below any land type
>>> polygon.
>>> Is that possible?
>>>
>>> Gerd
>>>
>>> ________________________________________
>>> Von: Pinns UK <osm at pinns.co.uk>
>>> Gesendet: Montag, 9. Dezember 2019 16:17
>>> An: Gerd Petermann; mkgmap-dev at lists.mkgmap.org.uk
>>> Betreff: Re: AW: AW: [mkgmap-dev] New branch for default typ file
>>>
>>> Hi Gerd
>>>
>>> Yes, I suppose so
>>>
>>> On 09/12/2019 15:14, Gerd Petermann wrote:
>>>> Hi Nick,
>>>>
>>>> my understanding is that you always have another water polygon,
>>>> either ocean or natural=water.
>>>>
>>>> Gerd
>>>>
>>>> ________________________________________
>>>> Von: Pinns UK <osm at pinns.co.uk>
>>>> Gesendet: Montag, 9. Dezember 2019 16:04
>>>> An: Gerd Petermann; mkgmap-dev at lists.mkgmap.org.uk
>>>> Betreff: Re: AW: [mkgmap-dev] New branch for default typ file
>>>>
>>>> Hi Gerd
>>>>
>>>> In case of 2) you need 2 polygons for doing each job; one showing
>>>> 'water' and the other one not
>>>>
>>>> Ideally,    mkgmap checks if islands are in a 'bay' area
>>>>
>>>> In my area we have lots of natural=bays ; fortunately they do not
>>>> include islands
>>>>
>>>> On 09/12/2019 14:51, Gerd Petermann wrote:
>>>>> Hi,
>>>>>
>>>>> thanks for the help.
>>>>> I see two ways to handle the a polygon with natural=bay:
>>>>> 1) in ponts style with natural=bay & name=*  [....]
>>>>> 2) in polygons (as it is now) with natural=bay [0x3d resolution
>>>>> 18]
>>>>>
>>>>> In case of 1) we just need option add-pois-to-areas In case of
>>>>> 2) we
>>>>> would want to render the water area covered by the bay polygon
>>>>> different, but not anything on the land or on islands. Would
>>>>> that be possible?
>>>>>
>>>>> Gerd
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
>>>>> Auftrag
>>>>> von Pinns UK <osm at pinns.co.uk>
>>>>> Gesendet: Montag, 9. Dezember 2019 15:42
>>>>> An: mkgmap-dev at lists.mkgmap.org.uk
>>>>> Betreff: Re: [mkgmap-dev] New branch for default typ file
>>>>>
>>>>> Andrzej is correct about how transparency is defined
>>>>>
>>>>> Garmin regards all polygons with transparency  as bitmaps and
>>>>> therefore require 2 colours.
>>>>>
>>>>> The Bitmap need to be shown below the xpm
>>>>>
>>>>> If a polygon is completely transparent then a second 'dummy'
>>>>> colour
>>>>> is still needed
>>>>>
>>>>> Xpm="32 32 2 1"
>>>>> "0 c none"
>>>>> "1 c #C8C8C8"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> "00000000000000000000000000000000"
>>>>> ;12345678901234567890123456789012
>>>>> [end]
>>>>>
>>>>> On 09/12/2019 14:19, Andrzej Popowski wrote:
>>>>>> Hi Gerd,
>>>>>>
>>>>>> I use TypViewer for creating typ files and I don't know XPM
>>>>>> details.
>>>>>> But looking at TypViewer output, I guess that transparent
>>>>>> pixels
>>>>>> are defined with color like that:
>>>>>>
>>>>>> "  c none"
>>>>>>
>>>>>> where space ' ' is used for marking pixels.
>>>>>>
>>>>>> Changing draw order instead of transparent graphics could be
>>>>>> a
>>>>>> solution too, but I'm not sure if covered polygon label would
>>>>>> remain visible. And without label, there is not much use of
>>>>>> this object.
>>>>>>
>>>>> _______________________________________________
>>>>> mkgmap-dev mailing list
>>>>> mkgmap-dev at lists.mkgmap.org.uk
>>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20191214/8fe72fe4/attachment-0001.html>