<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Dear Sirs:</p>
<p>Re UTF-8</p>
<p>Microsoft does this to handle another problem: Microsoft uses
many different character set encodings (i.e., code pages, e.g.,
CP1252) and the BOM is used by Microsoft to indicate that the
"code page" is UTF-8. Java was implemented to an older version of
the Unicode standard that prohibited an UTF-8 BOM. The problem is
comes from moving back and forth across that cultural divide. Yes,
this is painful.</p>
<p>A solution to the reading issue from the Java side:</p>
<p><a
href="https://stackoverflow.com/questions/1835430/byte-order-mark-screws-up-file-reading-in-java">https://stackoverflow.com/questions/1835430/byte-order-mark-screws-up-file-reading-in-java</a></p>
<p>A solution for writing a UTF-8 BOM in Java:</p>
<pre class="lang-java prettyprint prettyprinted" style="margin: 0px 0px 1em; padding: 12px 8px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; vertical-align: baseline; box-sizing: inherit; width: auto; max-height: 600px; overflow: auto; background-color: rgb(239, 240, 241); border-radius: 3px; display: block; color: rgb(57, 51, 24); overflow-wrap: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"><code style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; vertical-align: baseline; box-sizing: inherit; background-color: rgb(239, 240, 241); white-space: inherit;"><span class="typ" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(43, 145, 175);"><span style="color: rgb(36, 39, 41); font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(239, 240, 241); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(the File), StandardCharsets.UTF_8))</span></span><span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);"></span><span class="pln" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);">
out</span><span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);">.</span><span class="pln" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);">write</span><span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);">(</span><span class="str" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(125, 39, 39);">'\ufeff'</span><span class="pun" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: inherit; font-size: 13px; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54);">);</span></code></pre>
<p>A check for execution in a Windows environment:</p>
<p>String OS = System.getProperty("os.name").toLowerCase();<br>
Boolean isWindows = OS.indexOf("win") >= 0;</p>
<p>Perhaps, on output write the BOM in a Windows environment and use
the BOM optional on input.<br>
</p>
<p><a href="http://www.unicode.org/faq/utf_bom.html">http://www.unicode.org/faq/utf_bom.html<br>
</a></p>
<p class="q" style="color: rgb(96, 96, 96); font-family:
"Trebuchet MS", Arial, Georgia, sans-serif;
background-color: rgb(250, 250, 250); font-weight: bold;
font-style: italic; border: 1px solid rgb(217, 217, 217);
padding-bottom: 4px; padding-top: 4px; text-indent: -1.3em;
padding-left: 1.55em; font-size: medium; font-variant-ligatures:
normal; font-variant-caps: normal; letter-spacing: normal;
orphans: 2; text-align: start; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration-style: initial; text-decoration-color:
initial;">Q: What are some of the differences between the UTFs?</p>
<p class="a" style="color: rgb(0, 0, 0); font-family:
"Trebuchet MS", Arial, Georgia, sans-serif;
background-color: rgb(255, 255, 254); margin-left: 16px;
margin-right: 16px; font-size: medium; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration-style: initial; text-decoration-color:
initial;">A: The following table summarizes some of the properties
of each of the UTFs. </p>
<div style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 254); text-decoration-style:
initial; text-decoration-color: initial; font-size: 12.8px;">
<table class="faq" style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); border-collapse: collapse;
padding: 0.2em; margin-left: 1em; font-size: 1em;">
<tbody>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Name</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-8</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-16</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-16BE</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-16LE</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-32</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-32BE</th>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">UTF-32LE</th>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Smallest code point</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">0000</td>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Largest code point</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="right">10FFFF</td>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Code unit size</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">8 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">16 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">16 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">16 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">32 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">32 bits</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">32 bits</td>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Byte order</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">N/A</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center"><BOM></td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">big-endian</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">little-endian</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center"><BOM></td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">big-endian</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">little-endian</td>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Fewest bytes per
character</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">1</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">2</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">2</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">2</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
</tr>
<tr>
<th style="border-width: 1px; border-style: solid;
border-color: rgb(160, 160, 160); font-weight: bold;
padding: 5px; text-align: left; color: rgb(128, 128, 128);
background-color: rgb(248, 248, 248);">Most bytes per
character</th>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
<td style="color: rgb(0, 0, 0); font-family: Arial, Geneva,
sans-serif; background-color: rgb(255, 255, 254);
border-width: 1px; border-style: solid; border-color:
rgb(160, 160, 160); padding: 5px; text-align: left;"
align="center">4</td>
</tr>
</tbody>
</table>
</div>
<p class="a" style="color: rgb(0, 0, 0); font-family:
"Trebuchet MS", Arial, Georgia, sans-serif;
background-color: rgb(255, 255, 254); margin-left: 16px;
margin-right: 16px; font-size: medium; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration-style: initial; text-decoration-color:
initial;">In the table <BOM> indicates that the byte order
is determined by a byte order mark, if present at the beginning of
the data stream, otherwise it is big-endian.</p>
<div class="moz-cite-prefix"><a
href="http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf">http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Table 2-4. The Seven Unicode Encoding
Schemes<br>
<br>
Encoding Scheme Endian Order BOM Allowed?<br>
<br>
UTF-8
N/A yes<br>
<br>
The remainder of the table omitted.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><a
href="https://docs.microsoft.com/en-us/windows/win32/intl/using-byte-order-marks">https://docs.microsoft.com/en-us/windows/win32/intl/using-byte-order-marks</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">
<h1 id="using-byte-order-marks" style="box-sizing: inherit;
font-size: 2.5rem; margin: -10px 0px 0px; padding: 0px;
font-weight: 600; line-height: 1.3; overflow-wrap: break-word;
word-break: break-word; color: rgb(23, 23, 23); font-family:
"Segoe UI", SegoeUI, "Segoe WP",
"Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;">Using Byte Order Marks</h1>
<ul class="metadata page-metadata" data-bi-name="page info"
dir="ltr" style="box-sizing: inherit; margin: 0px !important;
padding: 0px !important; list-style: none; display: flex; color:
var(--text-subtle); font-size: 0.875rem; flex-wrap: wrap;
font-family: "Segoe UI", SegoeUI, "Segoe
WP", "Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial;" lang="en-us">
<li class="displayDate" style="box-sizing: inherit; margin: 0px;
padding: 0px; outline: 0px; list-style: none; display: flex;
flex-basis: auto;"><time role="presentation"
datetime="2018-05-31T00:00:00.000Z"
data-article-date-source="ms.date" style="box-sizing:
inherit;">05/30/2018</time></li>
<li class="readingTime" style="box-sizing: inherit; margin: 0px;
padding: 0px; outline: 0px; list-style: none; display: flex;
flex-basis: auto;">2 minutes to read</li>
<li class="contributors-holder" style="box-sizing: inherit;
margin: 0px; padding: 0px; outline: 0px; list-style: none;
display: flex; flex-basis: auto;"><a class="has-inner-focus"
href="https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/Intl/using-byte-order-marks.md"
title="2 Contributors" style="box-sizing: inherit;
background-color: transparent; color: var(--primary-base);
cursor: pointer; text-decoration: none; overflow-wrap:
break-word; outline: 0px;">
<ul class="contributors" data-bi-name="contributors"
style="box-sizing: inherit; margin: 0px !important;
padding: 0px !important; list-style: none; display:
inline-block;">
<li style="box-sizing: inherit; margin: 0px; padding: 0px;
outline: 0px; list-style: outside none circle; display:
inline-block; vertical-align: top;"><img
src="https://github.com/Karl-Bridge-Microsoft.png?size=32"
data-src="https://github.com/Karl-Bridge-Microsoft.png?size=32"
role="presentation" style="box-sizing: inherit;
border: 0px; height: 16px; max-width: 100%; display:
inline-block; border-radius: 100%; margin-top: 5px;
overflow: hidden; width: 16px;"></li>
<span> </span>
<li style="box-sizing: inherit; margin: 0px; padding: 0px;
outline: 0px; list-style: outside none circle; display:
inline-block; vertical-align: top;"><img
src="https://github.com/msatranjr.png?size=32"
data-src="https://github.com/msatranjr.png?size=32"
role="presentation" style="box-sizing: inherit;
border: 0px; height: 16px; max-width: 100%; display:
inline-block; border-radius: 100%; margin-top: 5px;
overflow: hidden; width: 16px;"></li>
</ul>
</a></li>
</ul>
<p style="box-sizing: inherit; margin: 1rem 0px 0px; padding: 0px;
overflow-wrap: break-word; color: rgb(23, 23, 23); font-family:
"Segoe UI", SegoeUI, "Segoe WP",
"Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-size: 16px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;">Always prefix a Unicode plain text file with a byte
order mark, which informs an application receiving the file that
the file is byte-ordered. Available byte order marks are listed
in the following table. Because Unicode plain text is a sequence
of 16-bit code values, it is sensitive to the byte ordering used
when the text is written.</p>
<div class="alert is-info" style="box-sizing: inherit; display:
block; position: relative; border-radius: 6px; font-size: 16px;
padding: 1rem; margin-top: 1rem; background:
var(--info-background) !important; color: rgb(23, 23, 23);
transition: height 0.5s ease-in 0s, opacity 0.5s ease-in 0s;
overflow-wrap: break-word; word-break: break-word; border: 1px
solid var(--info-background-glow-high-contrast); font-family:
"Segoe UI", SegoeUI, "Segoe WP",
"Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-style: initial; text-decoration-color: initial;">
<p class="alert-title" style="box-sizing: inherit; margin: 0px;
padding: 0px; max-width: 100%; overflow-wrap: break-word;
font-weight: 600; color: var(--info-dark);"><span class="docon
docon-status-error-outline" style="box-sizing: inherit;
font-family: docons; font-size: inherit; speak: none;
display: inline-block; font-style: normal; font-weight: 400;
font-variant: normal; text-transform: none; text-align:
center; direction: ltr; line-height: 16px;
-webkit-font-smoothing: antialiased;"></span><span> </span>Note</p>
<p style="box-sizing: inherit; margin: 1rem 0px 0px; padding:
0px; max-width: 100%; overflow-wrap: break-word;">A byte order
mark is not a control character that selects the byte order of
the text.</p>
</div>
<p style="box-sizing: inherit; margin: 1rem 0px 0px; padding: 0px;
overflow-wrap: break-word; color: rgb(23, 23, 23); font-family:
"Segoe UI", SegoeUI, "Segoe WP",
"Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-size: 16px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;"> </p>
<div class="table-scroll-wrapper" style="box-sizing: inherit;
overflow: auto hidden; color: rgb(23, 23, 23); font-family:
"Segoe UI", SegoeUI, "Segoe WP",
"Helvetica Neue", Helvetica, Tahoma, Arial,
sans-serif; font-size: 16px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;">
<table style="box-sizing: inherit; border-collapse: collapse;
border-spacing: 0px; table-layout: auto; width: 608px;
font-size: 0.875rem;">
<thead style="box-sizing: inherit;"><tr style="box-sizing:
inherit;">
<th style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: bottom; border-width:
0px 0px 1px; text-align: left; font-size: 0.875rem;">Byte
order mark</th>
<th style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: bottom; border-width:
0px 0px 1px; text-align: left; font-size: 0.875rem;">Description</th>
</tr>
</thead><tbody style="box-sizing: inherit;">
<tr style="box-sizing: inherit;">
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">EF BB BF</td>
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">UTF-8</td>
</tr>
<tr style="box-sizing: inherit;">
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">FF FE</td>
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">UTF-16, little endian</td>
</tr>
<tr style="box-sizing: inherit;">
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">FE FF</td>
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">UTF-16, big endian</td>
</tr>
<tr style="box-sizing: inherit;">
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">FF FE 00 00</td>
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">UTF-32, little endian</td>
</tr>
<tr style="box-sizing: inherit;">
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">00 00 FE FF</td>
<td style="box-sizing: inherit; padding: 12px 16px;
border-style: solid; border-color: var(--border);
display: table-cell; overflow-wrap: break-word;
line-height: 1.5; vertical-align: top; border-width: 1px
0px 0px; text-align: left;">UTF-32, big-endian</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 12/14/2019 3:41 AM, Ticker Berkin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1576316510.1426.24.camel@jagit.co.uk">
<pre class="moz-quote-pre" wrap="">Hi Joris & Gerd
Great to see the typ-files now in trunk and all the work in updating
mapnik.txt to the current default style. Next week I plan to go through
"20191209 mapnik update.pdf" and comment on it and possible changes to
the default style.
Some other questions however:
How do you see mapnik.txt now being maintained; will it be as as simple
.txt file with patches being supplied in the same way as other source
files, or will it be regenerated from your translation spreadsheet and
other sources? I'd prefer the simple text file approach, but this might
allow changes into the file which make it incompatible with the tools
Joris uses to enhance it.
It is currently in UTF8 format, with an appropriate BOM at the start of
the file. I don't know how the java input libraries determine the
conversion rules to internal unicode, but this file should be
consistent with all the others that contain characters outside the
simple ansi 7-bit range (roadNameConfig.txt, default/inc/address)
It contains the statement:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">CodePage=65001
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">This is saying the output should be unicode, but the output should be
the same as the associated map.
Also the FID should be removed.
Regards
Ticker
On Tue, 2019-12-10 at 09:59 +0000, Gerd Petermann wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi Joris,
the file mapnik.txt says "Based on mkgmap default style version:
r4262"
Is it the right file?
reg. line type 0x0b: highway=motorway_link & (mkgmap:exit_hint=true |
mkgmap:dest_hint=*)
I want to look at the DestinationHook. If I got that right it should
be OK to have a zero-length road with that type to get the wanted
destination hint. In that case we don't have to care about rendering.
Gerd
________________________________________
Von: Joris Bo <a class="moz-txt-link-rfc2396E" href="mailto:jorisbo@hotmail.com"><jorisbo@hotmail.com></a>
Gesendet: Montag, 9. Dezember 2019 20:45
An: Development list for mkgmap; Gerd Petermann
Betreff: RE: [mkgmap-dev] New branch for default typ file
Hi All,
I don't think any changes needed in mkgmap itself. When the draworder
of bay is lower then water it will display correctly.
See attached new typ-file for correct usage.
Even better (but this is a change in default style): don't use
natural = bay in polygons but only in points for displaying as name.
Today I spent some time testing and repairing.
The mapnik.txt in branch mkgmap-default-typ-r4268 was pretty old and
also did not have the translations of all the languages anymore. It
also lost draworder of a lot of polygons which made the bay-problem
occur.
I did a complete recheck of the most recent default-style in: mkgmap
-r4386.zip and changed de typ-file accordingly.
I downloaded a full europe-latest from geofabrik today, builded it as
a big full europa map with the default style of r4386 and with
mkgmap r4386.jar No errors occured.
I think it’s up to date again but some review and comments are always
welcome.
See typ-file in attachement,
Kind regards,
Joris
-----Oorspronkelijk bericht-----
Van: mkgmap-dev <a class="moz-txt-link-rfc2396E" href="mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk"><mkgmap-dev-bounces@lists.mkgmap.org.uk></a> Namens Pinns
UK
Verzonden: maandag 9 december 2019 18:31
Aan: Gerd Petermann <a class="moz-txt-link-rfc2396E" href="mailto:gpetermann_muenchen@hotmail.com"><gpetermann_muenchen@hotmail.com></a>;
<a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
Onderwerp: Re: [mkgmap-dev] New branch for default typ file
Hi Gerd
Yes, you can do that with a draw level 1 higher than sea.
Draw orders are defined at the beginning of a (txt) typ file just
before the polygons
using the following format
Type=0x type number , draworder
It is good practice to sort the draworders , as that is how they
appear in a typ file
[_drawOrder]
Type=0x03,1
Type=0x28,1
Type=0x54,1
Type=0x01,2
Type=0x09,2
Type=0x4E,2
Type=0x10F1C,2
etc etc
[end]
I have no idea what the draworder for sea is , but just make it one
higher
On 09/12/2019 16:41, Gerd Petermann wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi Nick,
I don't want to cut out islands from bay polygons, I thought about
a proper typ for 0x3d which somehow marks "calmer water"
and a draw order that puts this above water and below any land type
polygon.
Is that possible?
Gerd
________________________________________
Von: Pinns UK <a class="moz-txt-link-rfc2396E" href="mailto:osm@pinns.co.uk"><osm@pinns.co.uk></a>
Gesendet: Montag, 9. Dezember 2019 16:17
An: Gerd Petermann; <a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
Betreff: Re: AW: AW: [mkgmap-dev] New branch for default typ file
Hi Gerd
Yes, I suppose so
On 09/12/2019 15:14, Gerd Petermann wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi Nick,
my understanding is that you always have another water polygon,
either ocean or natural=water.
Gerd
________________________________________
Von: Pinns UK <a class="moz-txt-link-rfc2396E" href="mailto:osm@pinns.co.uk"><osm@pinns.co.uk></a>
Gesendet: Montag, 9. Dezember 2019 16:04
An: Gerd Petermann; <a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
Betreff: Re: AW: [mkgmap-dev] New branch for default typ file
Hi Gerd
In case of 2) you need 2 polygons for doing each job; one showing
'water' and the other one not
Ideally, mkgmap checks if islands are in a 'bay' area
In my area we have lots of natural=bays ; fortunately they do not
include islands
On 09/12/2019 14:51, Gerd Petermann wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi,
thanks for the help.
I see two ways to handle the a polygon with natural=bay:
1) in ponts style with natural=bay & name=* [....]
2) in polygons (as it is now) with natural=bay [0x3d resolution
18]
In case of 1) we just need option add-pois-to-areas In case of
2) we
would want to render the water area covered by the bay polygon
different, but not anything on the land or on islands. Would
that be possible?
Gerd
________________________________________
Von: mkgmap-dev <a class="moz-txt-link-rfc2396E" href="mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk"><mkgmap-dev-bounces@lists.mkgmap.org.uk></a> im
Auftrag
von Pinns UK <a class="moz-txt-link-rfc2396E" href="mailto:osm@pinns.co.uk"><osm@pinns.co.uk></a>
Gesendet: Montag, 9. Dezember 2019 15:42
An: <a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
Betreff: Re: [mkgmap-dev] New branch for default typ file
Andrzej is correct about how transparency is defined
Garmin regards all polygons with transparency as bitmaps and
therefore require 2 colours.
The Bitmap need to be shown below the xpm
If a polygon is completely transparent then a second 'dummy'
colour
is still needed
Xpm="32 32 2 1"
"0 c none"
"1 c #C8C8C8"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
"00000000000000000000000000000000"
;12345678901234567890123456789012
[end]
On 09/12/2019 14:19, Andrzej Popowski wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi Gerd,
I use TypViewer for creating typ files and I don't know XPM
details.
But looking at TypViewer output, I guess that transparent
pixels
are defined with color like that:
" c none"
where space ' ' is used for marking pixels.
Changing draw order instead of transparent graphics could be
a
solution too, but I'm not sure if covered polygon label would
remain visible. And without label, there is not much use of
this object.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
mkgmap-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
<a class="moz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
mkgmap-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
<a class="moz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a>
_______________________________________________
mkgmap-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
<a class="moz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
mkgmap-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
<a class="moz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a></pre>
</blockquote>
</body>
</html>