7 A user’s guide to the Web2c system

Web2c contains a set of TeX-related programs, i.e., TeX itself, METAFONT, MetaPost, BIBTeX, etc. The original implementation was by Tomas Rokicki who, in 1987, developed a first TeX-to-C system adapting change files under Unix, which were primarily the work of Howard Trickey and Pavel Curtis. Tim Morgan became the maintainer of the system, and during this period the name changed to Web-to-C. In 1990, Karl Berry took over the work, assisted by dozens of additional contributors, and in 1997 he handed the baton to Olaf Weber. The latest result is Web2c Version 7.3, which was released in March 1999. Our version has some updates for the forthcoming new release, and identifies itself as 7.3.7

The Web2c 7.3 system runs on Unix, Windows 3.1, 9x/ME/NT/2K/XP, DOS, and other operating systems. It uses Knuth’s original sources for TeX and other basic programs written in web and translates them into C source code. Moreover, the system offers a large set of macros and functions developed to augment the original TeX software. The core TeX family components are:

bibtex

Maintaining bibliographies.

dmp

troœ to MPX (MetaPost pictures).

dvicopy

Produces modified copy of DVI file.

dvitomp

DVI to MPX (MetaPost pictures).

dvitype

DVI to human-readable text.

gftodvi

Generic font proofsheets.

gftopk

Generic to packed fonts.

gftype

GF to human-readable text.

makempx

MetaPost label typesetting.

mf

Creating typeface families.

mft

Prettyprinting METAFONT source.

mpost

Creating technical diagrams.

mpto

MetaPost label extraction.

newer

Compare modification times.

patgen

Creating hyphenation patterns.

pktogf

Packed to generic fonts.

pktype

PK to human-readable text.

pltotf

Property list to TFM.

pooltype

Display web pool files.

tangle

web to Pascal.

tex

Typesetting.

tftopl

TFM to property list.

vftovp

Virtual font to virtual property list

vptovf

Virtual property list to virtual font.

weave

web to TeX.

The precise functions and syntax of these programs are described in the documentation of the individual packages or of Web2c itself. However, knowing a few principles governing the whole family of programs will help you to benefit optimally from your Web2c installation.

All programs honor the standard GNU options:

--help   

print basic usage summary.

--verbose

print detailed progress report.

--version

print version information, then exit.

For locating files the Web2c programs use the path searching library Kpathsea. This library uses a combination of environment variables and a few configuration files to optimize searching the TeX directory tree. Web2c can handle more than one directory tree simultaneously, which is useful if one wants to maintain TeX’s standard distribution and local extensions in two distinct trees. To speed up file searches the root of each tree has a file ls-R, containing an entry showing the name and relative pathname for all files “hanging” under that root.

7.1 Kpathsea path searching

Let us first describe the generic path searching mechanism of the Kpathsea library.

We call a search path a colon- or semicolon-separated list of path elements, which are basically directory names. A search path can come from (a combination of) many sources. To look up a file “my-file” along a path “.:/dir”, Kpathsea checks each element of the path in turn: first ./my-file, then /dir/my-file, returning the first match (or possibly all matches).

In order to adapt optimally to all operating systems’ conventions, on non-Unix systems Kpathsea can use filename separators different from “colon” (“:”) and “slash” (“/”).

To check a particular path element p, Kpathsea first checks if a prebuilt database (see “Filename database” on page 69) applies to p, i.e., if the database is in a directory that is a prefix of p. If so, the path specification is matched against the contents of the database.

If the database does not exist, or does not apply to this path element, or contains no matches, the filesystem is searched (if this was not forbidden by a specification starting with “!!” and if the file being searched for must exist). Kpathsea constructs the list of directories that correspond to this path element, and then checks in each for the file being sought.

The “file must exist” condition comes into play with “.vf” files and input files read by TeX’s \openin command. Such files may not exist (e.g., cmr10.vf), and so it would be wrong to search the disk for them. Therefore, if you fail to update ls-R when you install a new “.vf” file, it will never be found. Each path element is checked in turn: first the database, then the disk. If a match is found, the search stops and the result is returned.

Although the simplest and most common path element is a directory name, Kpathsea supports additional features in search paths: layered default values, environment variable names, config file values, users’ home directories, and recursive subdirectory searching. Thus, we say that Kpathsea expands a path element, meaning it transforms all the specifications into basic directory name or names. This is described in the following sections in the same order as it takes place.

Note that if the filename being searched for is absolute or explicitly relative, i.e., starts with “/” or “./” or “../”, Kpathsea simply checks if that file exists.

7.1.1 Path sources

A search path can come from many sources. In the order in which Kpathsea uses them:

1. A user-set environment variable, for instance, TEXINPUTS. Environment variables with a period and a program name appended override; e.g., if “latex” is the name of the program being run, then TEXINPUTS.latex will override TEXINPUTS.
2. A program-specific configuration file, for example, a line “S /a:/b” in dvips’s config.ps.
3. A Kpathsea configuration file texmf.cnf, containing a line like “TEXINPUTS=/c:/d” (see below).
4. The compile-time default.

You can see each of these values for a given search path by using the debugging options (see “Debugging actions” on page 80).

7.1.2 Config files

Kpathsea reads runtime configuration files named texmf.cnf for search path and other definitions. The search path used to look for these files is named TEXMFCNF (by default such a file lives in the texmf/web2c subdirectory). All texmf.cnf files in the search path will be read and definitions in earlier files override those in later files. Thus, with a search path of .:$TEXMF, values from ./texmf.cnf override those from $TEXMF/texmf.cnf.

While reading the description of the format of the file texmf.cnf below, please also refer to appendix 11, starting on page 89, which lists the texmf.cnf file on the CD-ROM.

• Comments start with “%” and continue to the end of the line.
• Blank lines are ignored.
• A \ at the end of a line acts as a continuation character, i.e., the next line is appended. Whitespace at the beginning of continuation lines is not ignored.
• Each remaining line has the form:
  
    variable[.progname] [=] value
  where the “=” and surrounding whitespace are optional.
• The “variable” name may contain any character other than whitespace, “=”, or “.”, but sticking to “A-Za-z_” is safest.
• If “.progname” is present, the definition only applies if the program that is running is named progname or progname.exe. This allows different flavors of TeX to have different search paths, for example.
• “value” may contain any characters except “%” and “@”. The “$var.prog” feature is not available on the right-hand side; instead, you must use an additional variable. A “;” in “value” is translated to “:” if running under Unix; this is useful to be able to have a single texmf.cnf for Unix, MSDOS and Windows systems.
• All definitions are read before anything is expanded, so variables can be referenced before they are defined.

A configuration file fragment illustrating most of these points is shown below:

7.1.3 Path expansion

Kpathsea recognizes certain special characters and constructions in search paths, similar to those available in Unix shells. As a general example, the complex path, ~$USER/{foo,bar}//baz, expands to all subdirectories under directories foo and bar in $USER’s home directory that contain a directory or file baz. These expansions are explained in the sections below.

7.1.4 Default expansion

If the highest-priority search path (see “Path sources” on page 65) contains an extra colon (i.e., leading, trailing, or doubled), Kpathsea inserts at that point the next-highest-priority search path that is defined. If that inserted path has an extra colon, the same happens with the next highest. For example, given an environment variable setting

Since it would be useless to insert the default value in more than one place, Kpathsea changes only one extra “:” and leaves any others in place: it checks first for a leading “:”, then a trailing “:”, then a doubled “:”.

7.1.5 Brace expansion

A useful feature is brace expansion, which means that, for instance, v{a,b}w expands to vaw:vbw. Nesting is allowed. This can be used to implement multiple TeX hierarchies, by assigning a brace list to $TEXMF. For example, in texmf.cnf, you find the following definition:

Using this you can then write something like

which means that, after looking in the current directory, the $HOMETEXMF/tex, $TEXMFLOCAL/tex, $VARTEXMF/tex and $TEXMFMAIN/tex trees only) will be searched (the last two use using ls-R data base files). It is a convenient way for running two parallel TeX structures, one “frozen” (on a CD-ROM, for instance) and the other being continuously updated with new versions as they become available. By using the $TEXMF variable in all definitions, one is sure to always search the up-to-date tree first.

7.1.6 Subdirectory expansion

Two or more consecutive slashes in a path element following a directory d is replaced by all subdirectories of d: first those subdirectories directly under d, then the subsubdirectories under those, and so on. At each level, the order in which the directories are searched is unspecified.

If you specify any filename components after the “//”, only subdirectories with matching components are included. For example, “/a//b” expands into directories /a/1/b, /a/2/b, /a/1/1/b, and so on, but not /a/b/c or /a/1.

Multiple “//” constructs in a path are possible, but “//” at the beginning of a path is ignored.

7.1.7 List of special characters and their meaning: a summary

The following list summarises the meaning of special characters in Kpathsea configuration files.

:

Separator in path specification; at the beginning or the end of a path it substitutes the default path expansion.

;

Separator on non-Unix systems (acts like :).

$

Variable expansion.

~

Represents the user’s home directory.

{...}

Brace expansion, e.g., a{1,2}b will become a1b:a2b.

//

Subdirectory expansion (can occur anywhere in a path, except at its beginning).

%

Start of comment.

\

Continuation character (allows multi-line entries).

!!

Search only database to locate file, do not search the disk.

7.2 Filename databases

Kpathsea goes to some lengths to minimize disk accesses for searches. Nevertheless, at installations with enough directories, searching each possible directory for a given file can take an excessively long time (this is especially true if many hundreds of font directories have to be traversed.) Therefore, Kpathsea can use an externally-built “database” file named ls-R that maps files to directories, thus avoiding the need to exhaustively search the disk.

A second database file aliases allows you to give additional names to the files listed in ls-R. This can be helpful to adapt to DOS-like “8.3” filename conventions in source files.

7.2.1 The filename database

As explained above, the name of the main filename database must be ls-R. You can put one at the root of each TeX hierarchy in your installation that you wish to be searched ($TEXMF by default); most sites have only one hierarchy. Kpathsea looks for ls-R files along the TEXMFDBS path.

The recommended way to create and maintain “ls-R” is to run the mktexlsr script included with the distribution. It is invoked by the various “mktex”. . .  scripts. In principle, this script just runs the command

If a file is not found in the database, by default Kpathsea goes ahead and searches the disk. If a particular path element begins with “!!”, however, only the database will be searched for that element, never the disk.

7.2.2 kpsewhich: Standalone path searching

The kpsewhich program exercises path searching independent of any particular application. This can be useful as a sort of find program to locate files in TeX hierarchies (this is used heavily in the distributed “mktex”. . .  scripts).

Kpathsea looks up each non-option argument on the command line as a filename, and returns the first file found. There is no option to return all the files with a particular name (you can run the Unix “find” utility for that).

The more important options are described next.

--dpi=num

Set the resolution to “num”; this only affects “gf” and “pk” lookups. “-D” is a synonym, for compatibility with dvips. Default is 600.

--format=name

Set the format for lookup to “name”. By default, the format is guessed from the filename. For formats which do not have an associated unambiguous suffix, such as MetaPost support files and dvips configuration files, you have to specify the name as found in the first column of Table 1, which lists currently recognized names, a description, associated environment variables¹⁰, and possible file extensions.

The last two entries in Table 1 are special cases, where the paths and environment variables depend on the name of the program: the variable name is constructed by converting the program name to upper case, and then appending INPUTS.

The environment variables are set by default in the configuration file texmf.cnf. It is only when you want to override one or more of the values specified in that file that you might want to set them explicitly in your execution environment.

Note that the “--format” and “--path” options are mutually exclusive.

--mode=string

Set the mode name to “string”; this only affects “gf” and “pk” lookups. No default: any mode will be found.

--must-exist

Do everything possible to find the files, notably including searching the disk. By default, only the ls-R database is checked, in the interest of efficiency.

--path=string

Search along the path “string” (colon-separated as usual), instead of guessing the search path from the filename. “//” and all the usual expansions are supported. The options “--path” and “--format” are mutually exclusive.

--progname=name

Set the program name to “name”. This can affect the search paths via the “.prognam” feature in configuration files. The default is “kpsewhich”.

--show-path=name

shows the path used for file lookups of file type “name”. Either a filename extension (“.pk”, “.vf”, etc.) or a name can be used, just as with “--format” option.

--debug=num

sets the debugging options to “num”.

7.2.3 Examples of use

Let us now have a look at Kpathsea in action.

The latter is a BIBTeX bibliography database for TUGBoat articles.

Next we turn our attention to dvips’s header and configuration files. We first look at one of the commonly used files, the general prolog tex.pro for TeX support, before turning our attention to the generic configuration file (config.ps) and the PostScript font map psfonts.map. As the “.ps” suffix is ambiguous we have to specify explicitly which type we are considering (“dvips config”) for the file config.ps.

We now take a closer look at the URW Times PostScript support files. The name for these in Berry’s font naming scheme is “utm”. The first file we look at is the configuration file, which contains the name of the map file:

It should be evident from these few examples how you can easily locate the whereabouts of a given file. This is especially important if you suspect that the wrong version of a file is picked up somehow, since kpsewhich will show you the first file encountered.

7.2.4 Debugging actions

Sometimes it is necessary to investigate how a program resolves file references. To make this feasible in a convenient way Kpathsea offers various debug levels:

• stat calls (file tests). When running with an up-to-date ls-R database this should almost give no output.
• References to hash tables (like ls-R database, map files, configuration files).
• File open and close operations.
• General path information for file types searched by Kpathsea. This is useful to find out where a particular path for the file was defined.
• Directory list for each path element (only relevant for searches on disk).
• File searches.

A value of -1 will set all the above options; in practice you will probably always use these levels if you need any debugging.

Similarly, with the dvips program, by setting a combination of debug switches, one can follow in detail where files are being picked up from. Alternatively, when a file is not found, the debug trace shows in which directories the program looks for the given file, so that one can get an indication what the problem is.

Generally speaking, as most programs call the Kpathsea library internally, one can select a debug option by using the KPATHSEA_DEBUG environment variable, and setting it to (a combination of) values as described in the above list.

(Note for Windows users: it is not easy to redirect all messages to a file in this system. For diagnostic purposes you can temporarily SET KPATHSEA_DEBUG_OUTPUT=err.log).

Let us consider, as an example, a small LaTeX source file, hello-world.tex, which contains the following input.

This little file only uses the font cmr10, so let us look how dvips prepares the PostScript file (we want to use the Type1 version of the Computer Modern fonts, hence the option -Pcms).

At this point dvips identifies itself to the user. . .

7.3 Runtime options

Another of the nice features of Web2c is its possibility to control a number of memory parameters (in particular, array sizes) via the runtime file texmf.cnf read by Kpathsea. The listing of texmf.cnf is shown in Appendix 11, starting on page 89; the settings of all parameters can be found in Part 3 of that file. The more important control variables are:

main_memory

Total words of memory available, for TeX, METAFONT and MetaPost. You must make a new format file for each different setting. For instance, you could generate a “huge” version of TeX, and call the format file hugetex.fmt. Using the standard way of specifying the program name used by Kpathsea, the particular value of the main_memory variable will then be read from texmf.cnf (compare the generic value and the “huge” one instantiated by hugetex, etc.).

extra_mem_bot

Extra space for “large” TeX data structures: boxes, glue, breakpoints, etc. Especially useful if you use PI CTeX.

font_mem_size

Number of words for font information available for TeX. This is more or less the total size of all TFM files read.

hash_extra

Additional space for the hash table of control sequence names. Approximately 10,000 control sequences can be stored in the main hash table; if you have a large book with numerous cross-references, this might not be enough. You can see that both the hugetex and pdÆatex program invocations ask for an extra 15,000 control sequences (the default value of hash_extra is zero).

Of course, this facility is no substitute for truly dynamic arrays and memory allocation, but since this is extremely difficult to implement in present TeX, these runtime parameters provide a practical compromise allowing some flexibility.