Roundup

Developing Roundup

Note

The intended audience of this document is the developers of the core Roundup code. If you just wish to alter some behaviour of your Roundup installation, see customising roundup.

Getting Started

Anyone wishing to help in the development of Roundup must read Roundup’s Design Document and the implementation notes.

All development is coordinated through two resources:

Website, wiki

The website is in our repository, so is the configuration of the wiki. Please check the README.txt in the “website” subdirectory of a current checkout.

Issue Tracker

The tracker resides on psf.upfronthosting.co.za. (2012-07-04) 2048 c4:ad:63:92:1b:c0:4a:35:c8:c5:3e:96:51:c0:88:67 (RSA)

The roundup installation belongs to the user roundup. In ~roundup, all trackers are stored and the roundup code itself. roundup is started through /etc/init.d/roundup; other parts of the installation are started through /etc/init.d/{postgresql-8-1,spambayes,postfix}.

The machine is operated by Upfronthosting in South Africa. The meta tracker is http://psf.upfronthosting.co.za/roundup/meta/ In this tracker, Upfronthosting people are the users izak and roche.

The Roundup tracker http://issues.roundup-tracker.org/ is in ~roundup/trackers/roundup

The configuration is in the “website/issues” section of Roundup’s Mercurical SCM repository and copied manually to the live tracker.

A checkout of the roundup sources is in ~roundup/src/roundup-src.

Small Changes

Most small changes can be submitted as patches through the issue tracker or sent to roundup-devel mailing list.

Source Repository Access

See http://www.roundup-tracker.org/code.html. For all other questions ask on the development mailinglist.

Project Rules

Mostly the project follows Guido’s Style (though naming tends to be a little relaxed sometimes). In short:

  • 80 column width code
  • 4-space indentations
  • All modules must have an Id line near the top

Other project rules:

  • New functionality must be documented, even briefly (so at least we know where there’s missing documentation) and changes to tracker configuration must be logged in the upgrading document.
  • subscribe to roundup-checkins to receive checkin notifications from the other developers with write access to the source-code repository.
  • discuss any changes with the other developers on roundup-dev. If nothing else, this makes sure there’s no rude shocks
  • write unit tests for changes you make (where possible), and ensure that all unit tests run before committing changes
  • run pychecker over changed code

The administrators of the project reserve the right to boot developers who consistently check in code which is either broken or takes the codebase in directions that have not been agreed to.

Internationalization Notes

How stuff works:

  1. Strings that may require translation (messages in human language) are marked in the source code. This step is discussed in Marking Strings for Translation section.
  2. These strings are all extracted into Message Template File locale/roundup.pot (POT file). See Extracting Translatable Messages below.
  3. Language teams use POT file to make Message Files for national languages (PO files). All PO files for Roundup are kept in the locale directory. Names of these files are target locale names, usually just 2-letter language codes. Translating Messages section of this chapter gives useful hints for message translators.
  4. Translated Message Files are compiled into binary form (MO files) and stored in locale directory (but not kept in the source code repository, as they may be easily made from PO files). See Compiling Message Catalogs section.
  5. Roundup installer creates runtime locale structure on the file system, putting MO files in their appropriate places.
  6. Runtime internationalization (I18N) services use these MO files to translate program messages into language selected by current Roundup user. Roundup command line interface uses locale name set in OS environment variable LANGUAGE, LC_ALL, LC_MESSAGES, or LANG (in that order). Roundup Web User Interface uses language selected by currently authenticated user.

Additional details may be found in GNU gettext and Python gettext module documentation.

Roundup source distribution includes POT and PO files for message translators, and also pre-built MO files to facilitate installations from source. Roundup binary distribution includes MO files only.

GNU gettext package

This chapter is full of references to GNU gettext package. GNU gettext is a “must have” for nearly all steps of internationalizing any program, and it’s manual is definetely a recommended reading for people involved in I18N.

There are GNU gettext ports to all major OS platforms. Windows binaries are available from GNU mirror sites.

Roundup does not use GNU gettext at runtime, but it’s tools are used for extracting translatable messages, compiling message catalogs and, optionally, for translating messages.

Note that gettext package in some OS distributions means just runtime tools and libraries. In such cases gettext development tools are usually distributed in separate package named gettext-devel.

Marking Strings for Translation

Strings that need translation must be marked in the source code. Following subsections explain how this is done in different cases.

If translatable string is used as a format string, it is recommended to always use named format specifiers:

_('Index of %(classname)s') % locals()

This helps translators to better understand the context of the message and, with Python formatting, remove format specifier altogether (which is sometimes useful, especially in singular cases of Plural Forms).

When there is more than one format specifier in the translatable format string, named format specifiers must be used almost always, because translation may require different order of items.

It is better to not mark for translation strings that are not locale-dependent, as this makes it more difficult to keep track of translation completeness. For example, string </ol></body></html> (in index() method of the request handler in roundup_server script) has no human readable parts at all, and needs no translations. Such strings are left untranslated in PO files, and are reported as such by PO status checkers (e.g. msgfmt --statistics).

Command Line Interfaces

Scripts and routines run from the command line use “static” language defined by environment variables recognized by gettext module from Python library (LANGUAGE, LC_ALL, LC_MESSAGES, and LANG). Primarilly, these are roundup-admin script and admin.py module, but also help texts and startup error messages in other scripts and their supporting modules.

For these interfaces, Python gettext engine must be initialized to use Roundup message catalogs. This is normally done by including the following line in the module imports:

from i18n import _, ngettext

Simple translations are automatically marked by calls to builtin message translation function _():

print _("This message is translated")

Translations for messages whose grammatical depends on a number must be done by ngettext() function:

print ngettext("Nuked %i file", "Nuked %i files", number_of_files_nuked)

Deferred Translations

Sometimes translatable strings appear in the source code in untranslated form [1] and must be translated elsewhere. Example:

for meal in ("spam", "egg", "beacon"):
    print _(meal)

In such cases, strings must be marked for translation without actual call to the translating function. To mark these strings, we use Python feature of automatic concatenation of adjacent strings and different types of string quotes:

strings_to_translate = (
    ''"This string will be translated",
    ""'me too',
    ''r"\raw string",
    ''"""
    multiline string"""
)
[1]In current Roundup sources, this feature is extensively used in the admin module using method docstrings as help messages.

Web User Interface

For Web User Interface, translation services are provided by Client object. Action classes have methods _() and gettext(), delegating translation to the Client instance. In HTML templates, translator object is available as context variable i18n.

HTML templates have special markup for translatable strings. The syntax for this markup is defined on ZPTInternationalizationSupport page. Roundup translation service currently ignores values for i18n:domain, i18n:source and i18n:target.

Template markup examples:

  • simplest case:

    <div i18n:translate="">
     Say
     no
     more!
    </div>

    this will result in msgid "Say no more!", with all leading and trailing whitespace stripped, and inner blanks replaced with single space character.

  • using variable slots:

    <div i18n:translate="">
     And now...<br/>
     No.<span tal:replace="number" i18n:name="slideNo" /><br/>
     THE LARCH
    </div>

    Msgid will be: "And now...<br /> No.${slideNo}<br /> THE LARCH". Template rendering will use context variable number (you may use any expression) to put instead of ${slideNo} in translation.

  • attribute translation:

    <button name="btn_wink" value=" Wink " i18n:attributes="value" />

    will translate the caption (and return value) for the “wink” button.

  • explicit msgids. Sometimes it may be useful to specify msgid for the element translation explicitely, like this:

    <span i18n:translate="know what i mean?">this text is ignored</span>

    When rendered, element contents will be replaced by translation of the string specified in i18n:translate attribute.

  • i18n in TALES. You may translate strings in TALES python expressions:

    <span tal:replace="python: i18n.gettext('Oh, wicked.')" />
  • plural forms. There is no markup for plural forms in TAL i18n. You must use python expression for that:

    <span tal:replace="python: i18n.ngettext(
      'Oh but it\'s only %i shilling.',
      'Oh but it\'s only %i shillings.',
      fine) % fine"
    />

Extracting Translatable Messages

The most common tool for message extraction is xgettext utility from GNU gettext package. Unfortunately, this utility has no means of Deferred Translations in Python sources. There is xpot tool from Francois Pinard free PO utilities that allows to mark strings for deferred translations, but it does not handle plural forms.

Roundup overcomes these limitations by using both of these utilities. This means that you need both GNU gettext tools and PO utilities to build the Message Template File yourself.

Latest Message Template File is kept in the source code repository and distributed with Roundup Source. If you wish to rebuild the template yourself, make sure that you have both xpot and xgettext installed and just run gmake (or make, if you are on a GNU system like linux or cygwin) in the locale directory.

For on-site i18n, Roundup provides command-line utility:

roundup-gettext <tracker_home>

extracting translatable messages from tracker’s html templates. This utility creates message template file messages.pot in locale subdirectory of the tracker home directory. Translated messages may be put in locale.po files (where locale is selected locale name) in the same directory, e.g.: locale/ru.po. These message catalogs are searched prior to system-wide translations kept in the share directory.

Translating Messages

Gettext Message File (PO file) is a plain text file, that can be created by simple copying roundup.pot to new .po file, like this:

$ cp roundup.pot ru.po

The name of PO file is target locale name, usually just 2-letter language code (ru for Russian in the above example). Alternatively, PO file may be initialized by msginit utility from GNU gettext tools:

$ msginit -i roundup.pot

msginit will check your current locale, and initialize the header entry, setting language name, rules for plural forms and, if available, translator’s name and email address. The name for PO file is also chosen based on current locale.

Next, you will need to edit this file, filling all msgstr lines with translations of the above msgid entries. PO file is a plain text file that can be edited with any text editor. However, there are several tools that may help you with this process:

  • poEdit by Vaclav Slavik. Very nice cross-platform GUI editor.

  • KBabel. Being part of KDE, it works in X windows only.

    At the first glance looks pretty hairy, with all bells and whistles. Haven’t had much experience with it, though.

  • po-mode for emacs. One of GNU gettext tools. Very handy, definitely recommended if you are comfortable with emacs. Cannot handle plural forms per se, but allows to edit them in simple text mode.

  • po filetype plugin for vim. Does not do as much as po-mode, but helps in finding untranslated and fuzzy strings, and checking code references. Please contact alexander smishlajev if you prefer this, as i have patched this plugin a bit. I have also informed the original plugin author about these changes, but got no reply so far.

Compiling Message Catalogs

Message catalogs (PO files) must be compiled into binary form (MO files) before they can be used in the application. This compilation is handled by msgfmt utility from GNU gettext tools. GNUmakefile in the locale directory automatically compiles all existing message catalogs after updating them from Roundup source files. If you wish to rebuild an individual MO file without making everything else, you may, for example:

$ msgfmt --statistics -o ru.mo ru.po

This way, message translators can check their PO files without extracting strings from source. (Note: String extraction requires additional utility that is not part of GNU gettext. See Extracting Translatable Messages.)

At run time, Roundup automatically compiles message catalogs whenever PO file is changed.