LyX wiki | LyX / Unicode

Categories: Unicode, Font, CJK, Japanese, Chinese

<< | Page list | >>

Instructions on setting up and using Unicode in LyX

Table of contents (hide)

1. Unicode
2. LaTeX source encoding
3. Unsupported unicode characters
4. CJK and Unicode
5. Devanagari and Unicode
6. Greek and Unicode
7. Math and Unicode

1. Unicode

http://en.wikipedia.org/wiki/Unicode

The Unicode Consortium: http://www.unicode.org/

2. LaTeX source encoding

Since version 1.5, LyX uses Unicode internally; the LyX file is always encoded in utf8.

The Menu Document>Settings>Language offers the Tab Encoding to specify the encoding of the LaTeX-file LyX produces. It is described in the UserGuide in Attachment A.6 "The Document Menu":

The encoding specifies how the document content is exported to LaTeX

LyX also supports Unicode output, which is particularly useful if you need lots of special symbols or non-alphabetic scripts, respectively. If you want to use this (and your LaTeX installation supports Unicode, for that matter), choose one of the four utf8 variants from the list below. Unfortunately the unicode support of standard LaTeX is quite incomplete, so it is not uncommon that a file with lots of unicode symbols works fine with use language's default encoding (when LyX uses it's list of known LaTeX-commands), but does not work with a fixed utf8 encoding (when the list of known LaTeX-commands is not used, because all unicode symbols can be encoded in utf8).

The variants of utf8-encoded LaTeX source are (Names changed in 1.6, 1.5-names in brackets):

[utf8] Unicode (utf8): Unicode utf8 based on the LaTeX-package inputenc. Currently only a limited range of characters (mainly for Latin scripts) is supported.
[utf8x] Unicode (ucs-extended) (utf8x): Unicode utf8 based on the LaTeX package ucs (comprehensive, including Latin, Greek, Cyrillic and CJK scripts).
[UTF8] Unicode (CJK) (utf8): Unicode utf8 with the LaTeX-package CJK (for the languages Chinese, Japanese and Korean)
[utf8-plain] Unicode (XeTeX) (utf8): Unicode utf8 to be used with XeTeX, which uses Unicode directly, without the help of the LaTeX-package inputenc

3. Unsupported unicode characters

If you come across a unicode character that is not yet supported by LyX, you can enter it here.

4. CJK and Unicode

Even though LyX 1.5.x says it supports CJK, it's a little tricky to get it working straight out of the box.

Material inserted 2009-01-22 <pauljohn@ku.edu>

I am inserting this material because the large effort described by previous author discouraged me from trying LyX with CJK and I don't want others to be similarly discouraged.

Use of Chinese in LyX-1.6 is simpler than the earlier writeup makes it seem. If you have LyX, LaTeX, and the CJK packages that come with your LaTeX distribution, the chances are that you have everything that is minimally necessary. Inspect your CJK installation and look for "tfm" ("tex font metric") files. These files indicate that you already have some Chinese fonts that LaTeX can use. In my system, in the directory "/usr/share/texmf/fonts/tfm/arphic" I found many files with names like this: gkaiu59.tfm . That indicates that the font "gkai" is available to latex (take the first 4 letters of the tfm file name). If you don't have the LaTeX fonts, well, get some. Its a tedious process.

Start LyX & run Tools/Reconfigure to make sure it takes note of your CJK LaTeX support. After closing LyX and re-starting in a clean document, a few changes are needed.

In Document/Settings/Language, change the language to Chinese (simplified, in my case) and change encoding to Unicode CJK (utf8).
In the Document/Settings/Fonts panel, change the CJK box to include a font you found.

You can write a document that includes English and Chinese characters, and when you process that document with latex (to get dvi) or "pdflatex" (to get pdf), LyX will "just work." LyX will handle the creation of the proper LaTeX preamble, there is no need for any ERT.

I'm working up a writeup on some of the difficulties that do exist in this CJK LyX framework, and I'm also working on a writeup on using XeTeX to do the same. But I don't want people to be left with the idea that a very complicated set of installations in needed to use Chinese.

Now, in the instructions below, one is encouraged to install new Chinese fonts, such as ipam or the Cyberbit fonts. These may make for nicer output, you will have to judge for yourself. As far as I can tell, there is no need whatsoever to install the Chinese ttf fonts because latex does not make any use of them. The truetype fonts might be useful in the XeTeX system, because it will make use of both truetype and opentype fonts. But in LaTeX, it is just not so.

I suppose I am saying the same thing as the author who says Note... the following "is clumsy," but I'm saying it even more emphatically and clearly. Most of it is not just clumsy, but unneeded, in LyX-1.6.

Here's the original longer more detailed writeup that has extra information. <pauljohn@ku.edu>

This page has a small tutorial on how to get it working. I have only had experience doing this on my Windows XP machine, but it's worked through all the 1.5.0 betas as well as the (as of now) current 1.5.1. As of now, this method does not work for Korean - but I suppose if someone was handy enough with creating packages of these tfms and fds it should be easy if you have a good truetype font.

Most of this info has come off of Jan Poland's helpful site. The instructions are more or less copied here. When I refer to "texmf" throughout this document, I refer to your main LaTeX folder - on my computer it's the Miktex folder in Program Files.

Note: The method described here is clumsy, because you have to use ERT in the document, or preamble definitions. If you install appropriate bitmap fonts (jsso12 for Japanese, cyberbit for Chinese). LyX will work out of the box at least for Japanese and Chinese. There's a step-by-step tutorial on how to install the cyberbit fonts here. I didn't find a tutorial for the jsso12 fonts. However, if you're running linux, installing the hbf-kanji48 package is sufficient.

a.) Install font metric files

Download this zip file found on Jan Poland's site or create your own using his instructions. Place them in texmf/fonts/tfm/IPA

b.) Download and install ipam.ttf

The Open Printing Project provides ipam.ttf for free in their package, which can be found here. It should be in the folder "ipafont". Place this file in texmf/fonts/truetype or a subdirectory.

c.) Install font definition files

Download this file and place the extracted files in a subdirectory of texmf/tex/latex - Alternatively you can create them using the instructions given.

d.) Find and edit ttfonts.map

Find ttfonts.map on your system and append the following to the end of the file:

ipam-uni@Unicode@ ipam.ttf

e.) Refresh file name database

Refresh your file name database - with Miktex it's accessed through the "settings" program.

f.) Create a test file

Now it's time to create a test file with some Japanese or Chinese text. First, make sure your document is set to utf8 encoding (found under Document Settings-> Language -> Encoding). Now that that's set, create two ERT blocks: the first should contain \begin{CJK}{UTF8}{ipam} . Inbetween that, place your Chinese or Japanese unicode text. Then, end it up with the ERT block \end{CJK} . That's it! Your file should create a nice little PDF with Japanese text (not antialiased, unfortunately).

An alternate method which does not require the ERT blocks was offered up by Marc Flerackers on the mailing list:

If you add the following preamble, you don't need to add the \begin{CJK}

\def\CJKhook{\CJKenc{UTF8}}
\ifx\CJKpreproc\undefined
  \def\CJKpreproc{cjk-enc}
  \RequirePackage[global]{CJK}
  \AtBeginDocument{\begin{CJK*}{UTF8}{ipam}\CJKspace}
  \AtEndDocument{\end{CJK*}}
\else
  \CJKhook
\fi

\usepackage[overlap, CJK]{ruby}

\CJKencfamily{UTF8}{ipam}

\renewcommand{\rubysep}{-0.3ex}

I have not personally tried this, but it supposedly works with mixed text documents. I may put up some test files once I get the password to upload with, if that would be helpful. I can also try to get permission from Jan to upload his files here.

5. Devanagari and Unicode

See this Wiki page.

Another (outdated page) about LyX, XeTeX and Devangari see here.

6. Greek and Unicode

http://wiki.lyx.org/LyX/Greek

7. Math and Unicode

See also for LyX 2.0 and later in XeTeX: http://wiki.lyx.org/LyX/XeTeX.

In LyX<=1.5.x, don't insert Unicode characters in Math fields: only standard latex commands will work.

This is a known limitation: http://bugzilla.lyx.org/show_bug.cgi?id=3938

A tricky example are symbols such as the greek letters (\alpha, ...), the centered dot or an ellipsis: when copied and pasted from other sources, the editor will not complain.

BUT, as soon as you try to compile the tex/ps/pdf output, the process will stop, most of the times without any error or warning. (See the threads on the lyx-users list http://thread.gmane.org/gmane.editors.lyx.general/47623/focus=47784, http://thread.gmane.org/gmane.editors.lyx.general/49830.) If you manually proceed step by step and convert the files yourself, you'll get an error like: "Basic::_M_convert_to_external call error", without reference on the source of the error.

An obvious solution is of course to use only non-multibyte characters in Math fields (plain ascii, except for text-in-math). You can make LaTeX recognise some Unicode chars with preamble commands, see http://article.gmane.org/gmane.editors.lyx.general/47784.

LyX 1.6.x supports Unicode chars in math. However, the default is to typeset anything except ASCII chars (and some symbols) as text-in-math. (see also http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg143678.html)

Category: Unicode Font CJK Japanese Chinese