The latest version of UTS #46 in Unicode 16 has changed the way it
indicates which codepoints are invalid in domain names, causing
'idna-mapping-table' to contain incorrect information, which then breaks
'textsec-domain-suspicious-p' and our test suite. (Bug#73312)
* admin/unidata/unidata-gen.el (unidata-gen-idna-mapping): Check the
IDNA validity field in "IdnaMappingTable.txt" in addition to checking
the status field, as the latter can now be 'valid' for disallowed
codepoints.
* lisp/url/url-cookie.el (url-cookie-write-file):
* lisp/international/titdic-cnv.el (tit-process-header):
* lisp/international/ja-dic-cnv.el (skkdic-convert):
* lisp/international/emoji.el (emoji--generate-file):
* lisp/emacs-lisp/loaddefs-gen.el (loaddefs-generate--rubric)
* admin/unidata/unidata-gen.el (unidata-gen-file)
(unidata-gen-charprop): Use the new functions.
* lisp/emacs-lisp/generate-file.el: New file to provide
convenience functions for generated files. It's not always
trivial to know which parts of the trailer that has to be
obfuscated to avoid not getting byte-compiled etc, and some parts
of the headers/trailers are usually forgotten when hand-coding
these.
* admin/unidata/unidata-gen.el (unidata-file-alist): Update the
default values of 'bidi-class' according to the latest Unicode
Standard.
* admin/notes/unicode: Mention possible changes in
DerivedBidiClass.txt that need to be reflected in unidata-gen.el.
* lisp/international/characters.el (#xfb50, #xfdf0): Fix the
Arabic block characters. (Bug#55256)
* admin/unidata/Makefile.in (${unidir}/uni-scripts.el): Build
uni-scripts.el.
* admin/unidata/Scripts.txt:
* admin/unidata/ScriptExtensions.txt:
* admin/unidata/PropertyValueAliases.txt: New files from Unicode.
* admin/unidata/README: Update.
* admin/unidata/unidata-gen.el (unidata-gen-charprop): Allow
writing other data, too.
(unidata-gen-scripts, unidata-gen--read-script-aliases)
(unidata-gen--insert-file): New functions to parse the Script* files.
* lisp/international/textsec.el: Implement some functions that
work on scripts.
* lisp/startup.el (keyboard-type): Make obsolete and lex-bound.
* admin/unidata/unidata-gen.el (unidata-gen-file)
(unidata-gen-charprop): Mark the generated files to use lexical binding.
* lisp/international/isearch-x.el: Use lexical-binding.
(junk-hist): Declare locally.
* lisp/international/iso-cvt.el:
* lisp/international/utf-7.el:
* lisp/international/robin.el:
* lisp/international/ogonek.el:
* lisp/international/latin1-disp.el:
* lisp/international/kkc.el:
* lisp/international/kinsoku.el:
* lisp/international/ja-dic-utl.el: Use lexical-binding.
* lisp/international/ja-dic-cnv.el: Use lexical-binding.
(skkdic-breakup-string): Remove unused var `kana-len`.
* lisp/international/latexenc.el: Use lexical-binding.
(tex-start-of-header): Declare.
* lisp/international/mule-diag.el: Use lexical-binding.
(list-character-sets): Remove unused var `pos`.
(list-character-sets-1): Remove unused vars `tail` and `charset`.
(list-charset-chars): Remove unused vars `chars` and `plane`.
(describe-coding-system): Remove unused var `extra-spec`.
(mule--print-opened): New var.
(print-fontset): Bind it.
(print-fontset-element): Use it instead of `print-opened`.
* lisp/international/quail.el: Use lexical-binding.
(quail-start-translation, quail-start-conversion):
Remove unused var `generated-events`.
(quail-help-insert-keymap-description): Use local dynbound var `the-keymap`.
(unidata-prop): Use defstruct to define the 6 accessor functions.
(unidata-gen-table-character, unidata-gen-table, unidata-gen-table-name)
(unidata-check): Move common code out of `if`.
(unidata-word-list-diff, unidata-split-decomposition):
Move common code out of `if`; use `push`.
* admin/unidata/copyright.html:
* admin/unidata/UnicodeData.txt:
* admin/unidata/SpecialCasing.txt:
* admin/unidata/NormalizationTest.txt:
* admin/unidata/Blocks.txt:
* admin/unidata/BidiMirroring.txt:
* admin/unidata/BidiBrackets.txt: New versions from Unicode 12.0.
* admin/unidata/unidata-gen.el (unidata-gen-file):
* admin/unidata/blocks.awk (name2alias): Adapt to changes in
new data files.
* admin/notes/unicode: Update and improve instructions for
importing a new Unicode Standard.
* lisp/international/characters.el (char-width-table): Update
lists of characters according to Unicode 12.0.
* lisp/international/fontset.el (script-representative-chars):
Add characters from new scripts to 'script-representative-chars'.
(otf-script-alist): Update according to data on the MS site.
* lisp/international/mule-cmds.el (ucs-names): Update unused
ranges of codepoints according to Unicode 12.0.
* test/lisp/international/ucs-normalize-tests.el
(ucs-normalize-tests--failing-lines-part1)
(ucs-normalize-tests--failing-lines-part2): Update for the new
NormalizationTest.txt file.
* test/manual/BidiCharacterTest.txt: Update with the new
version from Unicode 12.0.
I audited use of lsh in the Lisp source code, and fixed the
glitches that I found. While I was at it, I replaced uses of lsh
with ash when either will do. Replacement is OK when either
argument is known to be nonnegative, or when only the low-order
bits of the result matter, and is a (minor) win since ash is a bit
more solid than lsh nowadays, and is a bit faster.
* lisp/calc/calc-ext.el (math-check-fixnum):
Prefer most-positive-fixnum to (lsh -1 -1).
* lisp/vc/vc-hg.el (vc-hg-state-fast): When testing fixnum width,
prefer (zerop (ash most-positive-fixnum -32)) to (zerop (lsh -1
32)) (Bug#32485#11).
* lisp/emacs-lisp/bytecomp.el (byte-compile-lapcode):
Tighten sanity-check for bytecode overflow, by checking that the
result of (ash pc -8) is nonnegative. Formerly this check was not
needed since lsh was used and the number overflowed differently.
* lisp/net/dns.el (dns-write): Fix some obvious sign typos in
shift counts. Evidently this part of the code has never been
exercised.
* lisp/progmodes/hideif.el (hif-shiftleft, hif-shiftright):
* lisp/term/common-win.el (x-setup-function-keys):
Simplify.
* admin/unidata/unidata-gen.el, admin/unidata/uvs.el:
* doc/lispref/keymaps.texi, doc/lispref/syntax.texi:
* doc/misc/calc.texi, doc/misc/cl.texi, etc/NEWS.19:
* lisp/arc-mode.el, lisp/calc/calc-bin.el, lisp/calc/calc-comb.el:
* lisp/calc/calc-ext.el, lisp/calc/calc-math.el:
* lisp/cedet/semantic/wisent/comp.el, lisp/composite.el:
* lisp/disp-table.el, lisp/dos-fns.el, lisp/edmacro.el:
* lisp/emacs-lisp/bindat.el, lisp/emacs-lisp/byte-opt.el:
* lisp/emacs-lisp/bytecomp.el, lisp/emacs-lisp/cl-extra.el:
* lisp/erc/erc-dcc.el, lisp/facemenu.el, lisp/gnus/message.el:
* lisp/gnus/nndoc.el, lisp/gnus/nnmaildir.el, lisp/image.el:
* lisp/international/ccl.el, lisp/international/fontset.el:
* lisp/international/mule-cmds.el, lisp/international/mule.el:
* lisp/json.el, lisp/mail/binhex.el, lisp/mail/rmail.el:
* lisp/mail/uudecode.el, lisp/md4.el, lisp/net/dns.el:
* lisp/net/ntlm.el, lisp/net/sasl.el, lisp/net/socks.el:
* lisp/net/tramp.el, lisp/obsolete/levents.el:
* lisp/obsolete/pgg-parse.el, lisp/org/org.el:
* lisp/org/ox-publish.el, lisp/progmodes/cc-defs.el:
* lisp/progmodes/ebnf2ps.el, lisp/progmodes/hideif.el:
* lisp/ps-bdf.el, lisp/ps-print.el, lisp/simple.el:
* lisp/tar-mode.el, lisp/term/common-win.el:
* lisp/term/tty-colors.el, lisp/term/xterm.el, lisp/vc/vc-git.el:
* lisp/vc/vc-hg.el, lisp/x-dnd.el, test/src/data-tests.el:
Prefer ash to lsh when either will do.
Most of this change is to boilerplate commentary such as license URLs.
This change was prompted by ftp://ftp.gnu.org's going-away party,
planned for November. Change these FTP URLs to https://ftp.gnu.org
instead. Make similar changes for URLs to other organizations moving
away from FTP. Also, change HTTP to HTTPS for URLs to gnu.org and
fsf.org when this works, as this will further help defend against
man-in-the-middle attacks (for this part I omitted the MS-DOS and
MS-Windows sources and the test tarballs to keep the workload down).
HTTPS is not fully working to lists.gnu.org so I left those URLs alone
for now.
* admin/unidata/unidata-gen.el (unidata-check): Do not test special
casing mapping of characters since that mapping is not constructed from
the unidata.txt file.
Also, check for integer decoder and cons char earlier so that less
unnecessary processing is performed.
* admin/unidata/unidata-gen.el (unidata-gen-file):
Get Copyright line from copyright.html.
Put information in file header, not separate README.
(unidata-gen-charprop): Mention the source location.
* lisp/international/README: Remove file.
This is better for parallel builds, eg it eliminates race
conditions from having one process write multiple files.
* admin/unidata/Makefile.in (lparen, unifiles): New variables.
Parse unidata-gen.el, not charprop.el, to get the list of uni- files.
(all): Explicitly list the output lisp files.
(PHONY_EXTRAS): Remove.
(${unidir}/charprop.el): Change rule to just be for this file.
(${unifiles}): New rule to write each unicode lisp file.
(extraclean): Simplify.
* admin/unidata/unidata-gen.el (unidata-gen-charprop):
Quieten in batch mode.
(unidata-gen-files): Remove, no longer used.
* lisp/loadup.el: Update command-line parser.
* admin/unidata/unidata-gen.el (unidata-gen-file):
Make it work as a stand-alone function in batch mode.
(unidata-gen-files): Pass extra arguments to unidata-gen-file.
* admin/unidata/unidata-gen.el (unidata-file-alist):
Rename from unidata-prop-alist. All users changed.
Use file name rather than property name as the key.
(unidata-prop-prop): New function.
(unidata-prop-index, unidata-prop-generator, unidata-prop-docstring)
(unidata-prop-describer, unidata-prop-default, unidata-prop-val-list):
Change to parse the argument rather than unidata-prop-alist.
(unidata-gen-table-character, unidata-gen-table)
(unidata-gen-table-symbol, unidata-gen-table-integer)
(unidata-gen-table-numeric, unidata-gen-table-word-list)
(unidata-gen-table-name, unidata-gen-table-decomposition)
(unidata-gen-table-special-casing): Pass index as an argument.
(unidata-check): Adapt to unidata-file-alist.
Pass index to generator functions.
(unidata-gen-files): Adapt to unidata-file-alist.
Write each output file once only. Overwrite rather than delete.
Implement unconditional special casing rules defined in Unicode standard.
Among other things, they deal with cases when a single code point is
replaced by multiple ones because single character does not exist (e.g.
‘fi’ ligature turning into ‘FL’) or is not commonly used (e.g. ß turning
into SS).
* admin/unidata/SpecialCasing.txt: New data file pulled from Unicode
standard distribution.
* admin/unidata/README: Mention SpecialCasing.txt.
* admin/unidata/unidata-get.el (unidata-gen-table-special-casing,
unidata-gen-table-special-casing--do-load): New functions generating
‘special-uppercase’, ‘special-lowercase’ and ‘special-titlecase’
character Unicode properties built from the SpecialCasing.txt Unicode
data file.
* src/casefiddle.c (struct casing_str_buf): New structure for
representing short strings used to handle one-to-many character
mappings.
(case_character_imlp): New function which can handle one-to-many
character mappings.
(case_character, case_single_character): Wrappers for the above
functions. The former may map one character to multiple (or no)
code points while the latter does what the former used to do (i.e.
handles one-to-one mappings only).
(do_casify_natnum, do_casify_unibyte_string,
do_casify_unibyte_region): Use case_single_character.
(do_casify_multibyte_string, do_casify_multibyte_region): Support new
features of case_character.
* (do_casify_region): Updated to reflact do_casify_multibyte_string
changes.
(casify_word): Handle situation when one character-length of a word
can change affecting where end of the word is.
(upcase, capitalize, upcase-initials): Update documentation to mention
limitations when working on characters.
* test/src/casefiddle-tests.el (casefiddle-tests-char-properties):
Add test cases for the newly introduced character properties.
(casefiddle-tests-casing): Update test cases which are now passing.
* test/lisp/char-fold-tests.el (char-fold--ascii-upcase,
char-fold--ascii-downcase): New functions which behave like old ‘upcase’
and ‘downcase’.
(char-fold--test-match-exactly): Use the new functions. This is needed
because otherwise fi and similar characters are turned into their multi-
-character representation.
* doc/lispref/strings.texi: Describe issue with casing characters versus
strings.
* doc/lispref/nonascii.texi: Describe the new character properties.
* admin/unidata/UnicodeData.txt:
* admin/unidata/Blocks.txt:
* admin/unidata/BidiMirroring.txt:
* admin/unidata/BidiBrackets.txt: Update from Unicode 9.0.0beta.
* admin/unidata/unidata-gen.el (unidata-gen-files): Bind
'coding-system-for-read' to 'utf-8, as various Unicode data files
now actually use non-ASCII characters.
(unidata-setup-list, unidata-get-name): Support the new Tangut
Ideographs block.
* lisp/international/characters.el (standard-case-table): Add new
characters from Unicode 9.0.0.
(standard-category-table): Add Arabic block u+08A0..u+08FF. Add
Cyrillic Extended-C block.
(char-width-table): Update ranges per Unicode 9.0.0.
* lisp/international/fontset.el (script-representative-chars): Add
new scripts defined by Unicode 9.0.0.
(otf-script-alist): Add new OTF script tags.
* lisp/international/mule-cmds.el (ucs-names): Update ranges per
Unicode 9.0.0 additions.
* admin/unidata/unidata-gen.el (unidata-setup-list): Don't ignore
surrogates. This avoids assigning them the default
general-category of 'Cn', i.e. unassigned codepoints.
(unidata-get-name): Give surrogates synthetic names.
admin/unidata/unidata-gen.el (unidata-check): Bring this function up
to date with the currently supported methods of generating Unicode
property tables. Add a comment with a description how to invoke
the check. Update the copyright years in the reference to the
Unicode data files we use.
