Lilt Inc. FAQs

Lilt supports import and export of the formats listed in the table below. Moreover, export to both plain text and XLIFF 1.2 is available for any import format (see Download your work). All translated and confirmed segments can also be exported in TMX format for each Memory. Supported translation memory formats are TMX and SDLTM.

Desktop Publishing Formats

Extension

Format

docx, docm, dotx, dotm,pptx, pptm, potx, potm, ppsx, ppsm,xlsx, xlsm, xltx, xltm,vsdx, vsdm

Microsoft Office 2007+

ods, ots, odg, otg, odp, otp, odt, ott

OpenOffice / LibreOffice

pdf

PDF (import only!)

idml, mif

Adobe InDesign, FrameMaker

Microsoft OfficeThe older file formats doc, xls, and ppt support only text or xliff export, and will not support export to original format.

PDFfiles can only be imported as such and will be available for text export. Note also that PDF sometimes contains hard line breaks which can result in unfavorable segmentation. We suggest to either use the original file format, if available, or convert PDF to Word (there are lots of very good free tools for this) and translate that. Word can then save PDF again. In case you already started or even finished translating a PDF but require the original formatting, you can convert the PDF to Word and upload that to the same project. Exact matches from your PDF translation will then be autopropagated to this new document.

Text and Software Localization Formats

Extension

Format

txt, csv, tsv

Plain text, Comma- and Tab-separated Values

xml, htm, html, php

XML, HTML, PHP

md

Markdown

srt

SubRip subtitle file

json

Javascript Object Notation (JSON)

json+html

Javascript Object Notation (JSON) with embedded HTML

yml, yaml

YAML

strings

iOS/Mac Strings

properties

Java properties

wix, resx

Windows installer XML, .NET resources

lang

Skype language files

rdf

Mozilla RDF

dita, ditamap

DITA

Bilingual Formats

Extension

Format

xlf, xliff

XLIFF 1.2

mxlf, mxliff

Memsource XLIFF (undocumented format)

mqxliff

MemoQ XLIFF (undocumented format)

sdlxlf, sdlxliff

SDLXLIFF (undocumented format)

txml

Wordfast Pro TXML

ttx

Trados Tag Editor

Translation Memory and Termbase

Extension

Format

tmx, tmx.zip

Translation Memory eXchange

sdltm

SDL Trados Studio Translation Memory (import only)

tbx

TermBase eXchange (import only)

tsv

Tab-delimited text (import only)

csv

Comma-delimited text (import only)

xlsx

Microsoft Excel 2007+ (import only)

tmq

TestMaster (import only)

Maximum TMX file size: 200MB. For larger files, zip the TMX file and upload with the extension .tmx.zip.

For tsv, csv, and xlsx termbases, simply use a two-column format where the first column contains the terms in the source language and the second column the corresponding translations in the target language.

Packages

Extension

Format

sdlppx

SDL Trados Studio Project Package (import only)

SDLPPX packages can be imported in the Project Documents window. Lilt will upload the first document in the package, and will also automatically upload any associated SDLTM files into the Translation Memory.

Provide your Lilt PM with your webtranslateit project’s API key as described here

The connector will run on a schedule and check for new untranslated strings. For each locale that has untranslated strings a project in Lilt will be created. The untranslated strings will be added to those projects and a “lilt:in_progress” label will be added to the strings in webtranslateit

When the translations are complete they will be loaded back into webtranslateit and the “lilt:in_progress” labels will be removed.

Want to work with us? Here’s how it works:

Apply though our website here !

If you pass our initial screening, we’ll reach out to you when we have a project match and set you up with a test. We require content-specialized experience, so you’ll need to take a test for each content type that you work with.

If you pass the test, you’ll be cleared to work with us on a variety of projects, and you’ll enjoy the benefits of being part of our Liltlancer community!

Lilt allows clients to manage terminology: words or phrases that have pre-defined translations. Within the CAT Editor, Lilt has various features to assist translators and reviewers in adhering to terminology.

In the Editor

Terminology is highlighted in orange in the editor. This includes:

Source terminology highlighting.Source terms are highlighted in orange if the corresponding target term does not appear in the target. You can click on a highlighted source term to search it in the terminology sidebar and view all allowed translations (see section below).To show highlighted terms in the source, "Tags: Source"must be turned off in theSettings.

MT terminology suggestions.The MT system will display terminology suggestions in orange.

A "Missing Terms" tooltip is displayed to the right of a segment if there are missing terms in the target. It shows a suggested translation for each source term. A given term will show "(more options available)" there are multiple allowed translations for the same term. To view the other allowed translations, click the highlighted source term or search the source term in the Terminology orSearch Memorytabs.

If there are overlapping term entries (e.g. for "ancient Greece" and "Greece"), only the longest term is highlighted in the source and displayed in the tooltip (for "ancient Greece").

Terminology Sidebar

The Terminology sidebar is a way for users to access the Term Base, along with any metadata associated with the terms. Metadata typically is structured into relevant information for a given word regarding the part of speech, participle, case, as well as links to image context.

You can access this tool by selecting the sidebar option, and clicking the “Terminology” icon.

By default, the Terminology Search will populate with the full source segment which is active in the tool. You can modify this search as needed within the editable text box.

Searching Existing Terminology

Terms will appear in a list format with both source text and target text within the tool. Each of these items can be selected in order to view the metadata associated with it.

If you are searching for an item which has multiple variations within the Term Base, several options may appear in the target list. Depending on how thorough the Term Base uploaded into the Lilt Memory is, this functionality assists in selecting the best possible translation for the source content you are translating.

Adding New Terminology

This option is available to add any new terminology to the Term Base for future use and reference. Particularly useful when operating in teams, this is a powerful tool which will enable consistency.

When selecting this option in Lilt, you will be presented with an option to add the source text as well as the target text. Select the check mark to add the entry into the Term Base, or select “x” to discard.

Project Opportunities

We currently support translation between English and 40 different languages. We work with a variety of content types, from marketing, to user interface and technical hardware manuals.

We aim to provide our freelance translators with consistent, recurring work opportunities, so most of our customers are large companies with high-volume translation needs.

When we have a spot on a project team that matches your language pair, direction, and content specialization, a member of our team will reach out to confirm your interest in joining the project.

The Services Success Manager responsible for the project will then reach out to introduce you to the team, as well as provide assignment information and training materials.

Payments

Payments at Lilt are fast and convenient. We’ve partnered with Payoneer to release payments every Friday PST. Your payment covers any work you’ve done for Lilt since the last payout datemeaning you don’t have to wait for months after you’ve done the work to get paid for it.

Create a Project

To create a project navigate to the Project Dashboard and click on the New Projectbutton:

Kanban board

In the project creation window, choose a project name name, language combination, and optionally set a due date or share with your team:

Having clicked Next, you will see the second step: Create Project Step 2 Select a pre-existing memory or create a new one.

Click Createto create the project. Lilt will then open the Project view, which allows you to import documents. Lilt supports most major file formats, including language-industry specific formats such as XLIFF and Trados Studio Packages. You can import files by pressing UPLOAD FILEor by dragging files into the box:

If you wish to translate the same source files to multiple target languages, then you will need to create a new project for each language pair.

After selecting the language pair, Lilt will create a project and import the files for you. For larger files this process may take a few seconds to complete.

Project Management

The project dashboard lets you visualize the current status of your team's projects as a . Projects are displayed in the corresponding columns according to their current status and move between the columns whenever a status changes.

Each project card shows a quick summaryof the project, its statusand due date. The small icons at the bottom of a card give you the option to quickly perform actions such as archiving, deletingand sharing. If you click on the infoicon a detailed info pane appears on the right, which also allows you to edit a few project details and set/change a due date.

You can also move projects up and down within a column, e.g. to move high priority projects on top, or between two columns by a simple drag & drop:

Lilt can integrate with easyDITA via easyDITA localization jobs. This article describes what you need to enable the localization via the easyDITA API and how to create a localization job within easyDITA.

Lilt setup for easyDITA

In the EasyDita resource manager generate an API key and share this with your Lilt PM. They will take care of the setup on Lilt's side and arrange a schedule for the new documents to be gathered.

easyDITA Localization Jobs

The following steps should be taken in EasyDita to create a localization job that will be picked up by the Lilt connector.

Create a new localization job in easyDITA. Follow the steps in the easyDITA documentation here.

When the connector checks for new localization jobs it will create a project in Lilt for each locale in the localization job. If a locale doesn’t have a memory or isn’t supported in Lilt it will be skipped.

When the translation is complete in Lilt the content will be sent back to easyDITA. When the content has successfully been received by easyDITA the localization job will be marked as complete.

Lilt is a platform that equips businesses to optimize among speed, quality, and cost for large-scale localization projects.

Spence Green

The core technology is an interactive, adaptive machine translation systemthat learns in real-time from human feedback and/or existing translation memory data. Adaptation allows the system to progressively provide better suggestions to human translators, and higher quality for fully automatic translation.

Lilt is based on machine translation and translator productivity research at Stanford University and Google. Co-founders John DeNero and met while working on Google Translate in 2011, and started Lilt in early 2015 to bring the technology to modern businesses and translators.

Want to know more? We'd love to hear from you.

For questions about our translator community, or working as a translator, contact [email protected].

For help using the platform, contact [email protected].

Translator-focused technology

Lilt was built with you in mind. You can access our translation platform from a web browser anywhere in the world, and access is free for all qualified freelancers (Liltlancers!).

Lilt shows you predictive suggestions, which are custom-tailored to the source content and termbase. Every time you use the tool, its suggestions become more accurate for your translation style. We’ll never ask you to post-edit raw MT output!

“Lilt has achieved the unimaginable synthesis between a clean, versatile white sheet of paper and the resourcefulness and convenience of a CAT tool, plus the boost of powerful predictive typing. It’s the ultimate tool for our craftsmanship.”

-Georgina Reparado, Spanish Language Lead

Be part of a team

We believe that your best work is done when you have the support you need. Liltlancers are provided with the opportunity to have direct access to our customers and work in collaborative teams, so you can get your questions resolved quickly and focus on creating quality work.

Lilt’s engineers, project managers and sales team want to learn from you. You’ll have opportunities to influence our technology and improve your own experience with us by participating in surveys, case studies, and offering feedback to our team.

Liltlancer Benefits

Fast Payments: Waiting weeks for your compensation is unacceptable, and with Lilt, it’s a thing of the past. We pay translators within 3-5 days of project completion.

Product Support: Our support and translator success team is available to help with your questions 24/7.

Access to Lilt’s technology: Each Liltlancer is given a Lilt account, which can be used for whatever you like, including projects outside of your scope with Lilt.

Stable, recurring projects: Working with Lilt means there’s no new technology to learn with each customer. Our services team aims to make working on future projects as predictable as possible.

We designed Lilt to maximize translation productivity. The core technology is an interactive, adaptive machine translation system. In both research and commercial settings, we have observed that Lilt greatly increases translation throughput with no loss of translation quality.

This is a guide to get started as a:

Project Managers

This guide will take you through the steps involved in creating and managing a project in Lilt.

One advantage you will find in Lilt is that you can do project management and translation work in the same application. There is no need to install or sign up for an additional solution to manage TM resources and projects within teams. Project managers, translators and reviewers work seamlessly in the same app, just in different views.

1. Preparation/Migration

If you want to migrate an existing project, export it as xliff(preferred), sdlxliff, mqxliff, mxliff, etc.

If you want to start with a new project, just have your source/project files ready.

Export translation memoriesas tmx(preferred), sdltm, etc. which are required or could be useful (use *.tmx.zipfor large file uploads).

Export termbaseswhich are required or could be useful.

2. Create/configure a Lilt Team

Master user (billing account): add project managers and translators under Account settings > Team.

Project managers: add translators and reviewers under Account settings > Team.

3. Create new project in Lilt with a Lilt Memory

Create a new Memory

Add TMs and termbases to the Memory Note:You can add as many files as you want, but they should be related to your project's content.

Create the Project Note:You can also start with this step and create a new Memory with a new project or assign an existing Memory to it.

4. Add TMs and terminology

Ideally you have added your TM resources in the previous step, before creating a Lilt Project, so that:

You can see TM matches and build quotes accordingly from the very beginning.

Lilt can learn from all relevant data and provide proper suggestions from the very beginning.

You can add further TM resources at any time of the project.

5. Sharing and collaboration

If you have set up a Lilt Team already, you can simply press Share with Teamand all your project managers can see your project and then take care of assigning the work. Now you or your project manager(s) can share and assign translation and review work.

6. Manage projects

Finally, you will be able to manage your projects, view progress and deliver finished work.

Translators

Lilt's interface and user experience differ from conventional CAT tools. Change is hard. We know. But we have designed the system with the goal of making you productive in less than 10 minutes.The interactive tutorial in Lilt, which you see after signing up, should get you started. The articles in this guide will turn you into a power user.

To use Lilt, you need to understand the following concepts:

Memories- A Memory (also known as engineor model) is a collection of source/target sentences or terms for a specific language pair (e.g., English>French). It can be re-used across projects. It updates when:

Translation memory data is uploaded from file (e.g., from a TMX file)

A sentence is confirmed

Projects- A project is a collection of one or more documents. Each project is associated with exactly one Memory.

Documents- A documentis a file that contains source sentences to translate.

In order to get you started with translation just follow these steps:

1. Preparation

If you want to create your own project: Follow steps 1., 3. and 4. in the

If you were invited by a project managerto work on a project: You simply follow the link in the invitation e-mail or open the project from the project dashboard and then the document.

2. Translate and Review

When opening a document in a project you will be redirected to the CAT editor.

The editor integrates MT, TMs and TBs and offers standard functionality such as TM (fuzzy) matches, a lexicon with concordance search, manual segmentation and formatting tags adjustment, amongst other features.

You can manually review translations in the QA mode, run automatic QA checks and communicate with other translators, reviewers or project managers via comments.

3. Progress and delivery

When working on a document that was shared with you by a project manager, there is no need to constantly inform them about your status or deliver the final document when you are done. The project manager will see automatically your progress and can export the final document.

For your own projects or for checking in-between translation and formatting output (which we recommend to do at least once) you simply download the document.

Additional training resources

You can always sign up for our free live webinars or watch a recording for additional reference:

The CAT Editor (computer-aided translation) is the core of the Lilt platform and where translation happens. This article describes the different components that comprise the CAT Editor, including:

The Segment Editor

Editing Tools

Leaving Comments

Working with Tags

The Header (including configurable Editor settings)

The Footer

The Contextual Sidebar

Using Hotkeys

The Editor has two modes: Translate and Review. The two are largely the same, with a few exceptions; see this page for specific details about Review Mode.

Segment Editor

When you open the Editor, you will find your source text divided into numbered segments. An active source segment has three components:

The source text (ST)

The typing area for your target text (TT): you can type in your translation manually or insert a suggestion from the suggestion area. Alternatives for the next word to insert may appear in this area as ghost-words.

The suggestion area: TM matches above 75\% are shown first. If a match does not exist, then the interactive MT output appears. This section is interactive, and it updates as you type. Read more about Lilt and Translation Memory Matching.

Segment Editing Tools

There is a toolbar on the right-hand side of each segment. Its buttons change depending on the segment's activity and status.

In confirmed or unconfirmed segments that are active, you can refresh QA check, confirm the segment and access the Segment editing tools.

Using the Segment editing tools, you can:

Confirm a segment (Accept, in Review Mode)

Merge it with the next or previous segment

Split a segment

Copy the ST into the TT area (you may want to use this option when you come across a complex and difficult fragment in the ST and you want to translate it expression by expression without looking up at the ST area)

Insert the entire suggestion into the TT area (not available in Review Mode)

When a segment has already been confirmed but is not active at the moment, the toolbar allows you to edit tags (if available) or unconfirm the segment.

For inactive and unconfirmed segments, the toolbar will not be visible.

Comments

The Comments button is displayed on the right of each segment when it is active. If you are the only translator of a given document, you can use the option to add notes, to-do lists, etc. If there are more translators, you can use comments as a space for discussion. Read more on comments.

Tags

Lilt supports the import and export of tagged text. In the editor, you will see tagged spans in the source text. Most font styles (e.g., bold, italic, underline, color) will be rendered. Other arbitrary tags may not be shown although they will be preserved. When you are working with a tagged ST, translate the text as usual.

Tags show automatically after you confirm a segment or click on the </>tags icon in the segment. Lilt will automatically place the tags for you. You can manipulate the tag placement if you find errors or want to make changes. Hover over a tag to see its type/content, drag and drop in order to edit the placement. Read more on tags.

The header

The header bar gives you access to file options, settings, view options and tools.On the right side of the header bar you can see your average translation speed, measured in the number of words per hour. If you are translating from a source language that primarily uses characters, this counter will be replaced with characters per hour.

File

Allows you to downloadthe translated text as TXT, XLIFF or in the original format of the file (i.e. the format in which the file was uploaded). If some parts of the text have not been translated yet, they will be preserved and shown in the file in the original language.

View

Allows you to change the way the segments are displayed. You may choose to see only the confirmed ones, the unconfirmed ones, or all. You may also switch to the Split view, in which the TT is on the left and ST (or the area in which to type it) on the right, as opposed to the horizontal view.

Tools

Here, you can directly manage roles(such as a translator and a reviewer). Moreover, you can turn on a QA check manually.

Settings

This menu contains various ways to configure and customize the editing experience. These settings are set on a per-account basis, and any changes will persist across Documents and Projects.

X Segments per page in editor: Changes how many segments are displayed per page.

Sticky lexicon: Keeps the sidebar open after inserting a word from the Lexicon into the segment's target text.

Forward auto-propagation: Propagates any confirmed segment to other unconfirmed, 100\% Exact Matches throughout the entire document. In Review Mode, this applies to accepted segments instead of confirmed.

Backward auto-propagation: Propagates any confirmed segment and overwrites other confirmed, 100\% Exact Matches throughout the entire document. In Review Mode, this applies to accepted segments instead of confirmed.

Allow TAB key for autocomplete: Allows users to use the TAB hotkey to accept MT suggestions.

Display invisible characters: Displays a dot in place of invisible characters, such as spaces between words.

Replace last word of a prefix: When using the mouse to select an MT-suggested word, after initially typing part of that word, Lilt will append the remaining characters rather than insert the word as a whole.

Auto-scroll: Automatically scrolls to the next unconfirmed/unaccepted segment after confirming or accepting a segment.

Case-sensitive matching: Forces TM Matches to respect case-sensitivity.

Tags: all: Displays all Tags, including hidden Tags.

Tags: source: Always displays Tags in the segment source text.

Partial updates: Displays immediate, partial MT suggestions in the segment editor, when loading full MT suggestions could otherwise require more time (i.e. very long segments).

The footer

The footer bar allows you to:

See the name of the file you are working on and the language pair

Navigate segments and check how many have been confirmed

Check how many words have been translated out of the total number

See an estimate on how much time it will take to finish translating the document

The sidebar

You can use the tiles menu on the right to:

Access the Lexicon

View the target text in isolation using Target Document Preview

Use the Find and Replaceoption, which allows you to search for words and expressions in both ST and TT

Use the Memory Search option

Access the Termbase

Access the list of all the documents you have in a given project, switch between them, and see the details for each (the number of words and segments, the percentage of confirmed and reviewed segments, and the total translation time so far)

View any available metadata for the selected segment

See the cheatsheet

Hotkeys

The editor is designed to be operated entirely from the keyboard. The key interaction is autocomplete. Press Enterto insert the next highlighted word in the suggestion if you choose to accept it. Press Shift+Enterto insert the whole suggestion. Once you confirm your translation by pressing the check mark or Control+Enter, Lilt improves its suggestions for the rest of the segments.

This is the list of Hotkeys. You can access the cheatsheet anytime directly from the Editor, by choosing the question mark icon from the right-hand side tile menu. If the cheatsheet is not enough, there is a link to the full list at the top of it.

Windows

Mac

Action

Enter or Tab

Return or Tab

Insert the highlighted word

Shift + Enter

Shift + Return

Insert the complete suggestion

Control + Space

Shift + Tab

Select next word

Control + Shift + Space

+ Shift + Space

Insert non-breaking space

Control + Enter

+ Return

Confirm segment

Control + Down

+ Down

Go to next unconfirmed segment

Control + Up

+ Up

Go to previous unconfirmed segment

Control + Insert

+ I

Copy source to target

Control + Z

+ Z

Undo last change

Control + Alt + M

Toggle comments

Alt + T

Alt + T

Open lexicon with selected text

F7

F7

Run QA check on current segment

Esc

Esc

Close TM diff view

Shift + F3

Shift + F3

Toggle case of highlighted text

Lilt can integrate with Zendesk Articles, Categories, Sections, and Dynamic Content to automate the import and export of pages and other content for translation. This article describes:

How to set up a Lilt / Zendesk integration

How to sync and maintain the integration

The structure of exported documents back to Zendesk

FAQs

Note: You must be a project administrator to set up any integrations.

How to set up a Lilt / Zendesk integration

Sign in to Lilt. From your profile in the main menu, select Account, Integrations, and then Zendesk

Zendesk's support guide here

Fill out the appropriate information in the configuration screen.

Imported projects label: This is both the name of the Project that will be generated, and the name given to this specific connection.

Email of Zendesk account: The email used to sign into and maintain your Zendesk account.

URL: The URL of your organization's Zendesk instance (e.g. )

API Token: Lilt requires an API token to integrate with Zendesk. Learn more about generating a token from .

Polling Schedule: Lilt will check for new content based on the selected choice.

Target Languages: The languages you want to translate the documents into. The choices here are based on Memories you have already created.

How to sync and maintain the integration

Once an integration is created, it will be listed under Zendesk on the Integrations page based on the name in "Imported projects label."

Here you can:

Manually sync the data between Lilt and Zendesk. Lilt will immediately check for new content.

Edit the settings of the integration. You can also delete the integration here.

Exporting files back to Zendesk

Zendesk has a built-in model for accepting translated versions of documents, and Lilt will automatically export them back to the original document.

FAQs

Do I need to give Lilt my Zendesk credentials?

No. Lilt only needs your Zendesk email, and an API Token.

Will Lilt automatically import the images in my Zendesk articles?

Lilt imports content from Zendesk in JSON and HTML format, which does not include image files themselves.

Will Lilt automatically re-import content if it is changed?

Yes. Lilt checks for modified files in the designated Zendesk instance.

Introduction

You’ve been using Lilt, and are impressed with its translation capabilities. Now, just how good is Lilt’s translation? If you have some reference documents handy and a Python environment, you can self-measure the Lilt platform translation quality via the API, using an objective metric: the BLEU algorithm.

The BLEU Algorithm

Take a look at Wikipedia’s description of the BLEU algorithm:

BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is" this is the central idea behind BLEU. [...]

Scores are calculated for individual translated segmentsgenerally sentencesby comparing them with a set of good quality reference translations. [...]

BLEU's output is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1.

Requirements

You will need:

An active Lilt account with an API key

A Python environment with sacrebleu ( https://github.com/mjpost/sacreBLEU ) installed

A reference set of document(s) for which to calculate the average BLEU score across segments. See section “Preparing the test set” below for more information.

Procedure

Preparing the evaluation set

Select one or more reference documents as your evaluation set. Each segment in your evaluation set must have at least one human reference translation; preferably, each segment has multiple reference translations, which increases the robustness of the BLEU metric.

Documents in the evaluation set:

Should be representative of the type of text you usually translate

Must have high-quality human reference translations

Should contain 1000 - 3000 segments, as too small evaluation sets lead to unreliable metrics

Must not have been uploaded to or translated with Lilt, or any competing engines you are evaluating against

Generating Lilt output translations

We will be comparing Lilt’s output translation against human reference translations. In this guide, we assume the use of the API.

First, we must draw a distinction between adapted and unadapted machine translation models, as BLEU expectations differ.

Unadapted: The default models that are first created for a specific language pair when you create a project in Lilt using a default memory. They are pristine, in the sense that you have not done any translation with them or uploaded any TMX files. The expected BLEU scores for this unadapted model will be lower.

Adapted: Given a project, if you have uploaded TMX files or translated and confirmed segments within the project, the base model will have adapted to those translations. The expected BLEU scores for this adapted model will be higher.

Decide which type of model to generate BLEU scores against. We recommend doing both and comparing the BLEU scores of an unadapted and adapted model to get a sense of the quality increase that adaptation provides.

Choose a project

First, ensure you have a project created in Lilt in the language pair of the document: https://lilt.com/docs/api#tag-Projects.

[Adapted models only]: Use a memory on which a reasonable number of documents with content similar to your reference document have been translated. Alternatively, use a project with a standard memory that has been updated with a TMX file containing segments similar to the content in the reference document. This memory should have been given sufficient time to adapt to the TMX file.

Segment-by-segment

Run the translate endpoint on all segments in the reference document: https://lilt.com/docs/api#tag-Translate. No matter how you call the API, be sure you can later match the reference segments to the Lilt translation segments. This is essential during scoring.

Batch translation

You may also use batch translation instead of translating segment-by-segment. The translation output from Lilt, and therefore the resulting BLEU score, will be equivalent.

To do this, first upload a document: https://lilt.com/docs/api#operation--documents-post. Then, run pre-translation on it: https://lilt.com/docs/api#operation--documents-pretranslate-post and save the output segments.

Calculating the BLEU score against the reference translation

First, you must format the output and reference translations so that they can be easily processed with the Python package `sacrebleu`. Both output and reference translations:

Must in plain text format and UTF8-encoded

Must have one segment per line

Must perfectly align; that is, each segment in the reference file must match one-to-one on the same line as the corresponding segment in the translation output

It is possible to concatenate multiple output and reference translation files into a single file, provided that they fulfill the requirements above.

To calculate the BLEU score, run:

cat translated_segments.txt > sacrebleu [-tok zh] path/to/reference_segments.txt

Note that when running on Chinese or Japanese output, the optional flag [-tok zh] should be passed.

Details about advanced `sacrebleu` usage can be found at https://github.com/mjpost/sacreBLEU.

Caveats

The BLEU score is not the only metric of translation quality, and has its pitfalls. Namely, BLEU scores are compared to reference human translations, which differ from translator to translator. Therefore, BLEU scores give a general sense of how good translation is, but will never be a perfect assessment of translation quality.

Lilt supports the import, export, and placement editing of tagged text. This includes both within the CAT Editor itself, and in exported documents. This article describes:

Tags Overview

Editing Tags

Tags in Review Mode

Tags Overview

Lilt supports both structural tags (e.g. spacing, line breaks), and formatting tags (e.g. bold, italic, underline). Other arbitrary tags may not be shown, though they will be preserved.

Tags are not shown for unconfirmed segments, and can be ignored when translating text. The exception are formatting tags, which will be rendered in source segments.

click here

Editing Tags

Tags are first shown in the editor after you confirm a segment (note: this can be turned off by deselecting 'Show tags for confirmed' in the top menu and selecting Tools).

Once a segment has been translated and confirmed, Lilt will automatically place the source tags into the translated text. For more information on our Research Team's work on tag projection, .

You can manipulate tag placement through drag-and-drop. Simply mouse over the tag, click it, and drop it where the tag belongs.

A few notes about tag manipulation:

If you cannot see tags in the source segment, in the menu go to Settings and then select Tags: source: Always show source tags.

Manipulating tags does not unconfirm a segment.

Tags are automatically saved after being placed. No additional saving is required.

Clicking into the text of a confirmed segment will return you to the text editor, removing tags from view (this also does not unconfirm a segment). To view the tags again, click the </> button in the segment toolbar. You cannot modify both text and tags simultaneously.

Tags in Review Mode

Reviewers can manipulate tags and text the same way that Translators can modify confirmed segments. However, once a segment has been accepted, tags cannot be further manipulated unless the segment is unaccepted.

The Memory Search tool is a way for users to access information stored in the associated project Lilt Memory. This information is broken down into three main elements: Machine Translation suggestions, Translation Memory matches, and Termbase items. When you select a segment within the tool, the Memory Search function will automatically update. You can also search freely for any content by entering text within the editable text box.

You can access this tool by selecting the sidebar option, and clicking the “Memory” icon.

By default, the Memory Search will populate with the full source segment which is active in the tool. You can modify this search as needed within the editable text box.

Depending on how mature the project Lilt Memory is, the segment will be broken down into the following elements:

Machine Translation

This item appears in grey in the list, and shows the raw MT suggestion for the given word or segment. If there are no known matches for your segment, this will be the only item which appears on your list.

Translation Memory matches

This is the first item which will appear in orange on the list, and shows any previously translated content which provides a potential translation match, the percentage of matched content, and the user supplying the translation match. TM matches are always ordered according to percentage, with the highest matches appearing at the top.

Selecting a TM match from the sidebar will replace the pre-existing target text (if any) currently in the segment. This is true even when selecting an accepted segment in Review Mode.

Termbase

If there are any associated Termbases or glossaries uploaded into the Lilt Memory, these items will appear in this list on a word-by-word basis. These items will always have the original source-to-target translation.

When you upload a document, Lilt segments the source text with a view toward maximizing machine translation quality. MT quality tends to degrade with sentence length, so Lilt prefers shorter segments. However, the segmenter may make mistakes. You can correct them manually by splitting and merging segments. Note though that merging across paragraph boundaries or hard line breaks is not possible. Secondly, if you wish to merge with confirmed segment(s), you need to first unconfirm the segment(s) in question

Click the Split Segmentbutton at the right side of a source segment after opening the segment editing tools :

Then you can manually place split markers to divide the segment:

For merging two neighboring segments, simply click on Merge with previous segmentor Merge with next segment, if the option is available:

Note: Lilt cannot merge segments if the two segments in question are:

From different paragraphs in the source text

From two different cells in a spreadsheet

Two different trans-units in XLIFF files

Do I need to download and install Lilt?

No. Lilt is cloud-based software, so you need only login with your browser. Because it runs remotely, you probably want to know....

Is my data secure?

Yes. Communication between your computer and our database is encrypted. The data itself (e.g., source documents, translations, translation memories) are encrypted in our database.

Is my data shared with other users or third parties?

No. Your data is used to train personalized machine translation systems. Neither your data nor the trained systems are shared with or accessible by anyone other than you. We do not send your data to Google Translate, Microsoft Translator, or any other third-party service.

Do I need training to use Lilt?

No. When you register, a quick interactive tutorial will teach you the basics. The User Guide contains more advanced instructions. But it's short. Our design goal is for you to learn the system in five minutes or less.

I forgot my password!

You can request a password reset link on the login page. Click the "Forgot your password?" link on the bottom, enter your email address, and click on "Request password reset". An email with instructions including a password reset link for your account will be sent to you. If you don't receive the email within the next minutes, please check your spam folder.

Can I import my data to and from SDL Trados?

Yes. The User Guide shows you how to export and import via SDLXLIFF, SDLTM, and SDLPPX (i.e., SDL packages). Standard interchange formats such as XLIFF 1.2 and TMX are also supported.

Can I download my trained Memories / MT systems?

No. But you can always download the data (as TMX) from your Memories.

Do you use Google Translate or Microsoft Translator?

No. We develop and train our own machine translation systems. The open-source decoder we use is called Phrasal, which is distributed by the Stanford University NLP Group. We actively contribute to Phrasal. Our Lexicon/concordance service is also open source and is developed in conjunction with the UC Berkeley Oscii Lab. You can find it on Github.

As of August 29, 2019, Lilt currently supports 42 languages, as well as the capability to specify the appropriate regional variations:

Afrikaans

Arabic

Bengali

Bulgarian

Chinese

Croatian

Czech

Danish

Dari

Dutch

English

Farsi

Finnish

French

German

Greek

Hebrew

Hindi

Hungarian

Igbo

Indonesian

Italian

Japanese

Javanese

Korean

Norwegian

Pashto

Persian

Polish

Portuguese

Romanian

Russian

Slovak

Slovenian

Spanish

Swahili

Swedish

Thai

Turkish

Ukrainian

Urdu

Vietnamese

Language Pairs

English

Lilt supports translation between English and all other supported languages.

German

Lilt supports the translation of German into:

English

French

Italian

New Language Support

We are always adding new languages to the Lilt platform, typically based on three criteria:

Community and customer demand

Availability of training data

Baseline translation quality that is competitive with Google Translate and Microsoft Translator.

Email us at [email protected] if you would like to request that Lilt support additional languages.

Localization workflows are often fragmented and follow a waterfall model. Multiple tools and services (e.g. TMS, CAT Tool, MT API) are used and stitched together introducing risk and making it hard to manage the entire process. Moreover, translation happens in several stages, including pre-translation steps with TMs and/or MT, which leaves little control to the translator at the end of the chain. Feedback is not directly integrated but needs to be managed via separate synchronization steps (e.g. periodic TM syncs or MT re-trainings).

Lilt is a customizable toolkit for agilelocalization workflows. It integrates centralized TM, interactive, adaptive MT and cloud-based CATand can be accessed via a web app and API. The real-time feedback loopbetween the translation team and the Lilt system enables agile and continuous localization without extra human intervention or exchanging files between systems. This gives the client and the translation team more control over the entire translation workflow and thus mitigates risk.

When you use Lilt, your work is under your control.

Lilt's translation suggestions are generated using a combination of our parallel text and your personal translation resources. When you upload a translation memory or translate a document, those translations are only associated with your account. Translation memories can be shared across your projects, but they are not shared with other users or third parties.

Your data never leaves our data center. We don't use Google Translate, Microsoft Translator, Linguee, or any other third-party translation service. Every suggestion that you see originates from a service built and controlled by Lilt.

You can remove your data at any time or delete your account entirely. To delete your account, send us a message via the support chat window at the lower right of the browser or directly to [email protected]. Let us know if you have any questions about data or privacy.

More information

How Lilt Secures Your Data

Privacy Policy

Lilt is cloud-based software, meaning that there's no installation necessary and no software to manage; you just need a web browser and an internet connection. To sign into your account, click on SIGN INin the top right corner, enter your email address and password and then press the SIGN INbutton.

run the latest version

If you signed up with a Google account, choose SIGN IN WITH GOOGLE and then select your Google account that is associated with Lilt.

Although it's less common in the translation industry, cloud software is widely used in other knowledge-intensive fields such as law, software engineering, academia, and many more. Some benefits of cloud software are:

Platform independence -- Lilt can be used on any desktop or tablet computer with a browser and an internet connection (including iPads).

Continuous feature releases -- We are always adding new features to Lilt, which are immediately available to users without having to install or update anything.

Ease of support -- Our support team can solve issues with you in real-time without you leaving the application.

Translation quality -- Lilt provides higher-quality translations due to its tight integration with machine translation systems. This is made possible by being set up and distributed across multiple servers across the globe, opposed to MT systems deployed on a single desktop computer.

Your data is secured by enterprise-grade security measures and is never shared with other users or third parties.

Browser Support

Lilt is tested on the last two versions of Chrome, Safari, Firefox, and Internet Explorer (IE11 and Edge). Support for older browsers is unofficial. For security and performance reasons, we strongly recommend that you of your preferred browser.

Typically, machine translation (MT) is used to pre-translate texts that the human translator has to post-edit, i.e. correct the mistakes that the machine made. Lilt replaces post-editing with interactive and adaptivemachine assistance and provides an entirely different experience.

Memory

Interactive Adaptive MT

Lilt's core technology is an interactive, adaptive MT systemthat learns and adjusts in real-time from human feedback and/or existing translation memory (TM) data.

The unique characteristics and benefits of Lilt MT are:

Interactive: Interactive MT is more accurate because it can observe what the translator has typed and update the suggestions based on all available information. The user interacts with the MT within a sentence by accepting or rejecting words or phrases and thus triggering refinements of the MT suggestions for the remaining sentence.

Adaptive: Adaptive MT learns in real-time and allows the system to progressively provide better suggestions to human translators, and higher quality for fully automatic translation.

Real-time adaptation: Each new translation helps Lilt adapt both the Editor and the Lexicon to the translator's style and field of specialization. It also saves time by reducing the repetition of mistakes that were already corrected.

Customization: For best results, TMs should be uploaded to give Lilt a head start. This builds a custom MT system on top of our pre-trained baseline MT engines. Training the custom MT requires zero configuration and it continues to automatically train through the real-time adaptation - no manual syncs or re-trainingsneeded!

Integrated: A Lilt effectively integrates MT, TM and TB into one controlled interface that can be accessed via the web app or API. The Memory always provides the best suggestions based on all three components and indicates from which component the suggestion is coming from. This gives the human translator more control over the translation process - no pre-translationsteps!

Lilt can integrate with GitHub and GitLab to automate the import and export of files from a repository. This article describes:

How to set up a Lilt / GitHub or GitLab integration

How to sync and maintain the integration

The structure of exported files back to GitHub or GitLab

FAQs

Note: You must be a project administrator to set up any integrations.

How to set up a Lilt / GitHub or GitLab integration

Sign in to Lilt. From your profile in the main menu, select Account, Integrations, and then GitHub or GitLab

here

(for GitHub only) Input your GitHub username and personal access token.

Learn more about generating a personal access token from GitHub's support guide here.

Fill out the appropriate information in the configuration screen.

Repository: The URL of your repository (username/repo)

Source Branch: The branch that contains the documents to translate.

Target Branch: The branch where translated documents will be exported to.

Import folders: The names of the folders with documents to import.

Export folder: The names of the folder where translated documents will be exported to.

Imported projects label: This is both the name of the Project that will be generated, and the name given to this specific connection.

How to sync and maintain the integration.

Once an integration is created, it will be listed under GitHub or GitLab on the Integrations page based on the name in "Imported projects label."

Here you can:

Manually sync the data between Lilt and GitHub or GitLab. Lilt will immediately check for new content.

Edit the settings of the integration. You can also delete the integration here.

Exporting files back to GitHub & GitLab

Once files have been translated, Lilt will export them back to GitHub or GitLab in the folders that were specified in the configuration window.

Lilt will also append the translation language code as a suffix to the filename. For example, "hello.txt" that is translated into Spanish will become "hello_es.txt."

FAQs

Do I need to give Lilt my GitHub or GitLab credentials?

No. Lilt only needs the URL of the repository you want to translate, and a personal access token for GitHub integrations.

What type of files will Lilt automatically import? What happens if my folders have JPEGs in them?

Lilt will only import supported files for translation. It will ignore unsupported files, including images.

The full list of supported file filters in Lilt is located .

Will Lilt automatically re-import content if it is changed?

Yes. Lilt checks for modified files in the designated GitHub or GitLab repo.

Target document preview is a component of the Editor's context panel. It allows users to view the target text in context of its neighboring segments, without the accompanying source text, and is structured to appear like the source text of the original document.

How it works

To open the Target document preview window, open the context panel on the Editor's right-hand side, and then select the Target document preview icon.

Important notes:

Target document preview only displays the document's target text, and does not include any source text

Only confirmed segments will be shown

Any style formatting in the segment's source text (e.g. bold, italics) will be applied to the target text.

The target text will be structured to match the structure of the original document, including paragraphs, line breaks, etc.

Although Target document preview supports all file types, it does not display additional design structure beyond text (e.g. rows and columns of cells for spreadsheets, slides for presentations, etc.)

The preview window is interactive. While the window is open, selecting a segment in the Editor will force the preview window to scroll to that same segment. Selecting a segment in the preview window will do the same to the Editor.

Once each source document has been translated and/or reviewed, export the translated document using the document menu:

You can also export it directly from within the editor in the Filemenu:

Each document can be downloaded in the original format, plain text (TXT), or XLIFF.

If you click on the Deliverybutton on the sidebar menu in the project details page, you will see additional options:

You can export:

A project TMX file.

A project archive (ZIP) containing all of the translated documents.

A revision report that shows changes made by reviewers.

A reviewer can begin reviewing a document while a translator is working. You can open the Review mode by selecting Reviewin the drop-down menu next to the document name in your project or - if you are assigned as a reviewer - by just clicking on the document name. As a PM you can assign a separate reviewer from the Sharemenu or access the Review mode from the document menu.

QA features

The reviewer interface is the same Editor that is used for translation and you have access to all of its features. You may find our useful when reviewing. Press Accepton each segment to mark it as reviewed and lock the segment from further changes by the translator:

You can also accept all segments at once by clicking on the check mark next to Accept all confirmed segments on pageon top of the page.

Note: All non-TM matched segments are attributed to the Translator, even if a Reviewer makes edits in Review mode.

Lilt has tools and options to help you proofread your own translations or review other translators' work.

Automatic QA Checks

QA runs automatically after you confirm a segment. You can refresh it by clicking the QAbutton in each segment's toolbar or using the hotkey F7. To run the suite of automatic QA checks manually, click the Run QAoption in the Toolsmenu on the header:

English rules

If any possible issues are found, they will be highlighted in different colors.

red: spelling mistakes

yellow: grammar and punctuation mistakes

blue: stylistic mistakes/suggestions

Click on the highlighted words or phrases to see the found issue and suggestions for correction. After you have corrected the found mistakes, you can refresh the QA check to make sure there are none left.

You may also select Add to QA Rulesto add a term to your set of rules (terms) that the automatic QA should recognize as correct.

The QA check integrates LanguageTool which is an open-source proof-reading tool for many languages that integrates community-based rules for spelling, grammar and style. You can browse the rules on their website, e.g. the .

A few included example checks are (amongst many more!):

Misspellings

Doubled words

Doubled punctuation

Doubled whitespace

Leading/trailing whitespace

Missing/inserted numbers (both source and target)

Split view

Use the split view - a tabular, more compact layout of your document - for easier proofreading. You can toggle the split view in the Viewmenu.

The Lexiconcombines a glossary with a concordance. Both resources update while you work.

Highlight source terms to access the Lexicon:

When you click the lexicon icon, the Lexicon panel will expand from the right side of the window:

Concordance results are ordered according to the segment that you are translating. You can also type queries into the Lexicon text input box. You can add glossary entries by clicking the Add new entrybutton.

The general availability of the system can be checked on the following status page: status.lilt.com

In case of problems with Lilt, please contact support.

Update payment method

If you would like to update your payment method, please go to your accountsettings and select the Billingtab. Then click on Updateto change your credit card information.

A payment failed

Lilt uses Stripe to process credit card payments. We bill in dollars (USD) from the United States. Stripe runs a variety of fraud checks for every payment, and from time to time, a payment may fail through no fault of your own. Here are some common causes for payment failure:

Zip/postal code check failed

CVC / security code check failed

Cross-border transaction blocked by your bank

Missing PIN number for a debit card

Why does Lilt not always learn specific translations immediately?

Lilt's unique machine translation system combines large statistical models trained on millions of sentence pairs based on publicly available background data with your personal translation memories and confirmed translated segments in real time. Sometimes, the evidence of a specific translation coming from the background data outweighs your personal preference, and the system will continue to use the background translation. However, over time your preferred translations should be picked up. As an example, consider the English term bank. The background data contains thousands of translations into bancoin Spanish, but if you are translating a document on river erosion with the river bank vanishing over time, the system might not immediately be able to learn to prefer your choice of orilla(river bankin English) over banco. It may take multiple segments where you have to correct bancointo orillabefore the system is able to learn your translation as the one to be suggested in future segments of that document.

Alternatively, you can upload a termbase with the corresponding translations, or add a new termbase from scratch to the memory that is associated with your project. Go to Memories, click on the "Edit" button of the targeted memory (here English > Spanish), and click on "Add a new termbase". You can then edit the empty termbase (terminology.csv) and add the desired translations (e.g. bankand orilla).

Why is there a permanent spinning wheel? I can't confirm my segment!

In very rare cases, translation requests might get stuck (e.g. if there are a lot of users accessing the system). This is indicated by a spinning wheel below the segment number which blocks confirmation of your segment. You can use the filter to only show unconfirmed segments (Show Unconfirmedon the top right of the editor). It is advised to confirm all unconfirmed segments prior to the one with the spinning wheel. This will usually resolve the issue, and you can then confirm the current segment.

Certain source files often contain metadata, or context about certain text fields. This can include information describing what the text is (e.g. titles, names, synopses, etc.), or also additional information that can help translators understand more about the text (e.g. source images, web page URLs, supplemental pages, etc.).

If such metadata exists within the source file, Lilt will display it in the Segment metadata viewer, which is accessible within the Editor's right-hand context panel.

How it works

To view any segment metadata, open the context panel on the Editor's right-hand side, and then select the Segment metadata viewer icon.

You can also click on the green "Metadata available" icon within the segment itself.

Important notes:

Lilt looks for metadata based on common patterns within files, such as Excel cell locations, JSON keys, web page property types, etc. If your file is highly customized, or provides metadata in an alternate method, that metadata may not show up within Lilt

If there is no metadata available, Lilt will not display the green 'Metadata available' button. There is no need to check the context panel for each segment.

Lilt allows for multiple people to collaborate on translations by forming a Team. Within a Team, Project Managers share responsibility for managing content, workflows, and other Team members. They are also the ones who assign documents to others for translation and review.

This article describes:

How to Manage a Team

Assigning Projects

How to Manage a Team

Add members to your team by selecting Team from the main menu, and then Add Members.

There are two roles within a team:

Translator / Reviewerthe base role within Lilt, can:

translate or review documents assigned to them

Project Managerin addition to the above, can also:

view all Projects and Memories

create new Projects and Memories

modify existing Projects and Memories

assign documents

add and remove team members

set up connectors and integrations

Note: A user who has the role of Translator / Reviewer as part of one Team, can also be a Project Manager of their own individual Team.

Assigning Projects

To assign documents to Translators and Reviewers, open a Project, go to the desired document, and click the assign button ( ).

You can also select all the documents within the Project, and click Assign. The below dialog will appear.

You can assign both a translator, reviewer, and optional due dates.

To assign a translator and/or a reviewer, click 'Add translator' or 'Add reviewer'. Select a user from one of your existing Team members, or type in a new email address to invite that user to join Lilt and work on your project.

You can also assign documents while in Translate Mode by going to the main menu, selecting Tools > Assign document.

Note: Only one Translator and Reviewer can be assigned to a document at a time.

This article is a compilation of translation productivity tips.

How can I increase my productivity in the editor?

Don't focus on the suggestions.Typing the translation directly can often be faster than waiting for the suggestions to update. Consider the machine output during pauses while typing.

Ignore the machine suggestions if and when you want to.When you know the translation, and you're a fast typist, the machine assistance will only slow you down.

Post-edit if when you want to. Use the hotkey Shift+Enter to insert the full suggestion. This technique can be faster when you only want to make minor changes.

Learn the advanced 'hotkey' shortcuts; avoid using the mouse.

Learn the copy-source-to-target hotkey.The MT system will corrupt non-linguistic strings composed of numbers, equations, and punctuation.

For CJK input (Chinese, Japanese, Korean), we recommend the following tools:

Google Input Tools for Chrome

Pinyin Sougou

Configure your browser spell checker for the target language.

Try the Lexicon first for terminology.If you can't find what you need, switch to another resource. Keep in mind that tab switching is slow; try to avoid it.

How can I improve Lilt's suggestions?

Lilt is an adaptive system, which means that it needs data (specifically, source/target pairs) for adaptation. Here are some rules of thumb for improving translation quality:

Lilt works best when custom memories are created for each domain. You may have memories for software, legal, medical, etc. A common mistake that we observe is aggregating all data into the default memories. This limits the system's ability to adapt to any one domain.

20k segments / memoryseems to be the point at which people observe noticeable increases in translation quality. Uploading relevant TM data is the best way to improve suggestions for a domain.

Google Translate / Microsoft Translator are admittedly broader domain systemssince they are trained on more data. Baseline (i.e., unadapted) quality of Lilt is competitive for some domains, better for some domains (e.g., medical), and worse for some domains (e.g., software strings).

Fine-grained lexical distinctions for common words (e.g., "Party") are the hardest for the system to learn.It has likely seen that word millions of times in the training data; it will need to see your preference many times to learn the distinction reliably. You should find that it learns rarer words and phrases faster.

What is the most efficient way to review?

In the editor, confirmed segments are reduced in size so that more document context can fit on the screen. Clicking on a confirmed segment will unconfirm it for editing. You must confirm the segment again to save it. When translating long documents, you may accidentally unconfirm or skip a segment.

To filter out segments, go to the File Menu, select View, and then select the segments you want to view. You can use these filters to quickly locate untranslated/unsaved segments.

The QA Mode is also ideal for reviewing and making quick edits.

Lilt can integrate with Wordpress via the paid Wordpress Multilingual Plugin (WPML) to automate the import and export of web pages for translation. This article takes information from the WPML site, and describes:

How to activate and authenticate Lilt with WPML

Sending your content for translation

Monitoring translation progress

Returning translated content to Wordpress

FAQs

Note: You must be a project manager to set up any integrations. You will also need a Lilt API Token, which you can request from your customer success or service manager.

How to activate and authenticate Lilt with WPML

Login to your WordPress website, visit the WPML->Translation Management admin page, and click on the Translation Services tab.

Checking for completed translations

Scroll down to "I’m looking for translation management systems" and click the link.

Locate Lilt and click Activate button.

Once the service is activated, click the Authenticate button. You will be asked for your Lilt API Token (request one by emailing [email protected] )

A popup window will appear in which you can paste or enter your Lilt API Token. After entering the values, click the Submit button.

Lilt is now connected to your WordPress account, and you are now ready to send content for translation.

Sending your content for translation

To send pages for translation, go to the Translation Management module (WPML->Translation Management).

Select the pages you want to translate by clicking the checkboxes in the first column. Next, select the target languages by checking the boxes, and click the Add selected content to translation basket button.

Go to Translation Management and click on the blinking tab at the top of the page, which is called Translation Basket. On this tab, you will see a list of all the pages (jobs) that you added to the basket, as well as the languages these items are to be translated into.

Here, you can verify the content you wish to translate, remove pages you accidentally added, and change the Batch name. The Batch name is visible on the Translation Jobs tab and will help you find projects. Think of a batch as the technical scope of a purchase order. You can refer to the batch names when communicating with your translation provider, which they will also see. When you are ready, simply click the Send all items for translation button to send everything to Lilt.

Note: Lilt will create a Project for each batch that you send, for each language you want that batch translated into. Read more about Projects in Lilt.

When the translation basket’s contents have been successfully sent to Lilt, you will see a confirmation message, as shown in the following figure.

At this point:

If you are a Lilt Software-only customer, you can sign in to Lilt to find and manage all of your Projects as usual.

If you are a Lilt Services customer, no further action is required. Lilt will match qualified translators to work on your Projects, just like content that comes from any other source.

Monitoring translation progress

You can follow the translation progress of the documents you sent to Lilt from your translation jobs tab, or from the Lilt app itself.

To monitor progress from within Wordpress, click on the Translation Jobs link in the confirmation message or click on the Translation Jobs tab at the top of the screen. On the next administration screen, the Batch that you just sent to translation will appear.

To monitor progress from within Lilt, you can go directly to the Lilt website, or you can click on the batch name to be redirected to Lilt.

Returning translated content to Wordpress

Check for completed jobs by visiting Translation Jobs inside your WordPress admin -> WPML -> Translation Management.

Locate the batch you want to check and click on the Synchronize status button.

Depending on your configuration, translations either will be delivered automatically or can be fetched manually from your WordPress admin panel after you click the Synchronize status button.

To check your configuration settings, open WPML->Translation Management and click on the Multilingual Content Setup tab. Find the Translation Pickup mode section and check the selected option.

The option labeled Translation Service will deliver translations automatically using XML-RPC means that a translation will be delivered to your website automatically after you click the Synchronize status

The option labeled The site will fetch translations manually means that completed translations can be downloaded from the Translations Dashboard (WPML->Translation Management) by using the Check status and get translations button at the bottom of the page after clicking the Synchronize status

Once the jobs are fetched, their status in the Translation jobs tab will be changed to “Complete.”

FAQs

How do I know when my translations are complete?

If you are a Lilt Software-only customer, you can view the status of all your Projects within the Lilt Projects Dashboard.

If you are a Lilt Services customer, your Service Manager will send you an email notifying you that all translations are complete.

When you’re ready to import the translated pages back to Wordpress, simply go to the Translation Jobs tab and click the Synchronize status button next to the batch you want to fetch.

This will trigger a synchronization with Lilt servers and the jobs will be ready to download. Next, just follow the steps outlined under the section in the guide above.

Lilt can integrate with Contentful to automate the import and export of pages for translation. This article describes:

Links to help prepare a Contentful instance for localization

How to set up a Lilt / Contentful integration

How to import and export pages to and from Lilt

FAQs

Note: You must be a project manager to set up any integrations.

Links to help prepare a Contentful instance for localization

To localize pages on Contentful, you must first set up your site to support multiple languages, specifically called locales in Contentful. The following links can help:

Localization with Contentful

Locales

Editors, here’s how translation works with Contentful

Rich text and localization

Once you have set up localization, you will see separate areas in your page with dedicated space for your desired languages.

How to set up a Lilt / Contentful integration

The integration between Lilt and Contentful functions as a Contentful Extension. To add Lilt as an extension, select Settings from the Contentful navigation bar, and then Extensions. On this next page, select Add Extension, and then Install from GitHub.

In the GitHub URL field, paste the following link: https://github.com/lilt/lilt-contentful/blob/master/extension.json

You will be asked to supply the following fields:

A name for your extension (we recommend “Lilt”)

An API Token. To request yours, please contact your customer success or service manager

Your extension is now set up.

Next, select Content Model from the Contentful navigation bar, and then select the Content Type you want to localize. Select Sidebar, and then select Use Custom Sidebar. You will see Lilt listed as one of the available extensions. Add Lilt as a custom extension, and then save the Content Model.

How to import and export pages to and from Lilt

When opening a page based on a Content Type that uses Lilt as an extension, you will see additional Lilt functionality on the right-hand sidebar.

To send the current page to Lilt, click the ‘Translate All Languages’ button.

This sends your page to Lilt and performs the following operations:

Lilt creates a Project for each language of your page

On the page in Contentful, you will see the progress status of each translation project. These correspond to: in Backlog, in Progress, in Review, Done

When a Project is complete, the button will change to say “use translation” and will pull it back into the post, and populating your translated text in the appropriate fields

FAQs

What Contentful fields does Lilt support?

Lilt supports all plain text and rich text fields. This includes embedded text from other pages.

If I make edits to my page after it has been translated, how can I send the updated page to Lilt?

The “Translate All Languages” button will change to a “Send to Lilt” button if it has not already been completed.

Lilt can integrate with Google Drive to automate the import and export of documents for translation. This article describes:

How to set up a Lilt / Google Drive integration

How to sync and maintain the integration

The structure of exported documents back to Google Drive

FAQs

Note: You must be a project manager to set up any integrations.

How to set up a Lilt / Google Drive integration

Sign in to Lilt. From your profile in the main menu, select Account, Integrations, and then Google Drive.

Fill out the appropriate information in the 'Select Google Drive configurations' screen.

Import folders: Enter a link to your Google Drive folder that contains the documents you want to translate. This folder's settings must allow Lilt to access it.

Imported project label: This is the name of the Project that will be generated. One Project will be generated for each language selected in "Exported Languages."

Import schedule: Lilt will check for new content based on the selected choice.

Exported languages: The targeted languages you want to translate the documents into. The choices here are based on Memories you have already created, and you can select one Memory per target language.

How to sync and maintain the integration

Once an integration is created, it will be listed under Google Drive on the Integrations page based on the name in "Imported projects label."

Here you can:

Manually sync the data between Lilt and Google Drive. Lilt will immediately check the selected Google Drive folder for new content.

Edit the settings of the integration. You can also delete the integration here.

Exporting files back to Google Drive

Translated documents will be returned to the same Google Drive folder they were imported from. Lilt exports the files with the same file name, with the translated language appended to the front of the file name.

For example:

A document titled "emailtemplate.html" that has been translated using the language pair "en -> es" will be returned to Google Drive as a file titled "[en -> es] emailtemplate.html".

FAQs

What type of files will Lilt automatically import? What happens if I leave JPEGs in my folder?

Lilt will only import supported files for translation. It will ignore unsupported files, including images.

Will Lilt automatically re-import a file in Google Drive if it has been changed?

Yes. Lilt checks for modified files in the designated Google Drive folder.

Lilt can integrate with Slack so that localization managers, project managers, translators, and reviewers can all coordinate and communicate together. Localization teams use both tools today to quickly share updates, style guides, ask translation questions, and more.

This article describes:

How to integrate Lilt and Slack

Connecting Lilt Projects and Slack Channels

Leaving comments in Lilt and Slack

How to integrate Lilt and Slack

Sign in to Lilt. From your profile in the main menu, select Account> Integrations> Slack Comments in Lilt

You will be taken to Slack to authorize Lilt with your desired Slack workspace.

Connecting Lilt Projects and Slack Channels

Slack Channels can be associated to both new and existing Projects.

When creating a new Project, you can assign a Slack Channel to that Project by clicking on the Select Slack Channel button, and then selecting the desired Slack Channel

To assign a Slack Channel to an existing Project, select the more details "i" button, and then selecting the desired Slack Channel on the right.

Leaving comments in Lilt and Slack

Once a Project has been connected to a Slack Channel, comments in Lilt will automatically be sent to Slack. The comment will include:

Project Name

Document Name

Segment Number

Each item is a link in Slack that will take you back to the corresponding Project, Document, or Segment in Lilt.

To respond to comments, you can either leave a comment back in Lilt, or select from a set of default responses in Slack, which will be sent back to Lilt.

This KB Article describes more on .

Lilt integrates with the following technologies for easier continuous localization, file sharing, and communication.

API

For technical users who want real-time customizable integration with Lilt, see:

Lilt API docs

Content Connectors

For customers who want to seamlessly move data from their own systems into and out of Lilt, we offer a large number of connectors, including:

Adobe Experience Manager

Alfresco Digital Business Platform

Amazon S3

Bitbucket

Box Drive

Confluence

Contentful

Documentum

Dropbox

Egnyte

Episerver

IBM FileNet

GitHub

GitLab

Google Drive

LifeRay

Nuxeo

OneDrive

OpenText

Salesforce

SharePoint

Sitecore

Micro Focus

WordPress

Zendesk

We are always adding new content connectors. Let us know if there is a connector you would like at [email protected].

Slack Notifications

Send reviewer feedback and other updates from active Lilt projects to channels in your team's existing Slack workspace:

Slack Integration

The Editor allows you to add comments to segments. If you are the only translator of a given document, you can use the option to add notes, to-do lists, etc. If there are more translators, you can use comments as a space for discussion. The comments feature can be integrated with Slack.

Comments

You can leave comments for yourself, other translators, project managers, and reviewers by pressing the comment icon next to a segment. Comments are kept in a thread so that you can keep the history. They can be edited, deleted or resolved and contain clickable URLs, e.g. links to term definitions.

Slack

Slack

Lilt is integrated with, a collaboration tool for teams to connect in a common workspace, communicate via messages and share files and links with one another. This helps teams to stay coordinated and work faster. In a localization or translation setting, our Slack integration in Lilt can help you to:

share updates and information about a project with the entire team

share style guides with the entire team

share and resolve issues directly

ask questions directly to the client

discuss terminology

notify a PM, reviewer or client that work has finished

notify a translation team about new upcoming projects

just stay in touch

Connect Lilt with Slack

A connection with Slack can only be created by the master account in a business account. Other team members can then be added in Slack as usual.

Open Account> Integrations> Add to Slack.

Select the Slack team in the upper right corner and click Authorize. Select Slack team

3.Select the desired channel when creating a new project or in the project settings which you can access by clicking on the info icon in the project card.

Comments in Slack

Once the connection with Slack is done and a channel is selected for a project, comments from Lilt can be seen in Slack as well. A comment from Lilt always includes the project name, document name and segment number. You can click on each of them to get directly to the corresponding project, document or segment. You can either select a pre-defined answer from Slack (see the "Choose a response" drop-down) or comment in Lilt.

This article describes how to manage Memories in Lilt, including how to:

Create a Memory

Edit and Manage a Memory

Export a Memory

Create a Memory

To create and populate a Memory:

Select Memories from the main menu.

Select NEW MEMORY on the Memories page:

Give your Memory a Name and a Language Pair. Press Create.

Select your new Memory from the list of Memories.

Upload TM resources by pressing Import and selecting files from your local drive. Supported formats are TMX, SDLTM, TBX, CSV, TSV, SDLTM, and/or two-column XLSX format.

Edit and Manage a Memory

You can edit and modify a Memory at any time. This includes:

Importing additional Translation Memories (TM)

Importing additional Termbases (TB)

Exporting a Memory

Exporting a TB

Purging the Memory

Updating the Memory's source and target languages. (Note: Updating a Memory will also update the Projects that use current Memory)

To edit and modify a Memory:

Select Memories from the main menu.

Select your new Memory from the list of Memories.

Select the desired action from the list of options.

Exporting a Memory

To export a memory:

Select the Memories tab.

Select the desired memory.

Select Export memory as TMX.

Lilt allows users to upload Translation Memories (TM) to supplement its baseline Memory when providing translation suggestions through the interactive, adaptive engine. This article describes:

How Lilt provides its suggestions

TM Matching order of operations

How TMs are used in Pretranslation

How Lilt provides its suggestions

Lilt will automatically provide suggested translations the moment you select a segment in the CAT Editor. This suggestion is based on Lilt's baseline training data.

KB article on Pretranslation

If you have uploaded a TM to the associated Memory, Lilt will scan the TM for segments that have similar words. If it finds segments with similar words and phrases that meet a 75\% threshold (known as a Fuzzy Match), Lilt will display the TM's suggestion and describe it as such.

Lilt only shows one suggested phrase at a time. Users can open the Search Memory pane to view additional suggestions.

TM Matching order of operations

Lilt makes suggestions based on the following order of operations:

In-Context Exact Match, also known as an ICE Match (sometimes called a Guaranteed Match). Not only does the selected segment have the exact same words as a segment in the TM (an Exact Match), the segments before and after the selected segment are also identical as three consecutive segments in the TM. If there are multiple ICE Matches, the most recent is chosen.

Exact Matches. The selected segment has the exact same words as a segment in the TM. If there are multiple Exact Matches per a given source segment, the most recent is chosen. Note that Lilt does care about capitalization case sensitivity and punctuation differences.

Fuzzy Matches. The selected segment has a number of similar words and phrases as a segment in the TM (roughly 75\%), but there are significant differences.

If no better matching exists, Lilt then defaults to its Machine Translation suggestion (MT).

If the Memory Search pane is open, Lilt will display all the suggestions it has available.

How TMs are used in Pretranslation

When running pretranslation on documents, Lilt will only populate target text for segments that are ICE or Exact Matches to the selected TM (unless the option to use Machine Translation is selected). Lilt will follow the same order of operations as above when encountering segments with multiple TM matches.

See this for details on Lilt and Pretranslation.

Pretranslation is a powerful way to accelerate the translation process by having Lilt automatically translate known segments based on a document's associated Memory, specifically Exact and ICE matches. It can also be used as a preparatory tool to gauge how much of a document's text comprises new words.

This article describes:

How Pretranslation works

How to use Pretranslation

Learn more about Lilt and Translation Memories.

How Pretranslation works

When Pretranslating a document, Lilt will scan the entire document and populate segments that are Exact or ICE matches when compared to the Memory used by the Project. These translated segments will be considered confirmed and attributed to the user within the designated Memory.

If Pretranslation is run after any translation work has been done, Lilt will ignore any already-confirmed segments (regardless of whether the prior translation was done by pretranslation or a human translator).

How to use Pretranslation

To use pretranslation, find the desired document within a Project, and select Pretranslate. Select the three-dot menu on the right (...), and select Pretranslate. For multiple documents, select the Select All button, and then Pretranslate.

Select from the optional dialog options from the below screen.

Review

The descriptions are as follows:

Translate with machine translation service: Lilt will translate the entire document and populate each segment using machine translation, in addition to translating Exact or ICE matches from translation memory. Populated segments that are not Exact or ICE matches will not be considered confirmed.

Attribute translation authorship to: All translated segments will be attributed to the current user who selects "Pretranslate," regardless of attribution within the associated Memory.

Automatically accept exact matches. This pertains to "" mode. Reviewers must accept segments to officially mark that the segment has been reviewed. Selecting this option automatically accepts Exact or ICE matches. This is different from confirmed segments in "Translate" mode.

Use case sensitive translation memory: This option forces Lilt to only translate Exact or ICE matches that also follow the same letter casing.

Lilt organizes translation data into containers called Memories. This article describes describes the specifics of Memories and how they function, including:

What are Memories?

About the Default Memory

How Memories are Trained and Updated

Best Practices for using Memories

To learn more about managing Memories, see: How to Use Memories

What are Memories?

A Memoryis a collection of source/target sentences for a specific language pair (e.g., English>French). The data in the Memory is used to train the MT system, populate the Translation memory (TM), and update the termbase.

Memories are private to your accountthe data is not shared across users. However, as a project manager you can share a Memory with your team so that all project managers within that team can access and modify the Memory (e.g. upload TMs, edit termbases, etc.).

How to Use Memories

About the Default Memory

Lilt comes with a standard, Default Memory for each supported Language Pair. This memory is based on generally available data with no domain-specific suggestions (unless you import additional TMs or Termbase files into the Default memory).

All projects that are not associated with a specific Memory automatically share this same Default Memory.

How Memories are Trained and Updated

Memories in Lilt are automatically updated and trained whenever the translation of a segment is confirmed. Each confirmed segment helps Lilt adapt both the CAT Editor and the Lexicon to your style and field of specialization.

Best Practices for using Memories

Lilt works best when it can first learn from a TM that contains similar content to the text to be translated. You can create a separate custom memory, specializing the suggestions you see, for each client or for broader domains like legal contracts or IT documentation. The same intuitions that apply to organize traditional translation memories apply here. In fact, creating custom memories for each domain or client rather than aggregating all data into one (e.g. the default memory) can prevent latency issues and improve speed as well as improve the translation quality of the suggestions.

To learn more about creating custom memories or importing existing TMs, see:

Memories in Lilt contain an extensive terminology editor called a Termbase (TB). Words or phrases in the TB act as a glossary, which Lilt will use to make suggestions when those words or phrases appear during translation. Termbases act supplementary to Translation Memories and can significantly improve the quality and efficiency of your translations.

This article describes how to manage the Termbase, including:

How to import a Termbase

How to edit a Termbase

How to add metadata to a term

Import a Termbase

To import a Termbase (TB):

Select Memories from the main menu

Select a memory from the list of Memories

Under Manage Resources, Select Import

Upload your TB (supported formats are: TBX, CSV, TSV, and XLSX).

For TSV and XLSX files, the first two columns must be the source and target language terms.

If the first row of the TB file contains a header, or if your file has metadata (e.g. TBX), check the circle for Termbase file includes a header and metadata.

Edit a Termbase

In addition to uploading TB files, you can also manually add and edit the contents of a TB.

Select Memories from the main menu

Select Manage Termbase

In this interface, you can manually edit the terms of your TB, or add new terms altogether.

How to add metadata to a term

Each term in a termbase can have additional metadata in it that will be displayed in the CAT Editor. Metadata can provide useful context about each of the terms, such as links to webpages with product descriptions or images.

To add additional metadata to a term, select the View metadata button next to the desired term.

Lilt automatically populates each term with three pieces of metadata:

Author

Date created

Date updated

For each new piece of metadata, first define the Property (a description of the metadata), the Type, and then the Value.

There are three Types of additional metadata you can add:

Text, which allows you to type any value with no special properties

URL, which while create a clickable hyperlink to the field in Value

Checkbox, which allows you to select a checkbox, or leave it empty

If you have additional questions or would like to report a problem, please email us at [email protected]. You can also contact us through the chat window on the website.

Translation quality is subjectiveunlike, say, optical character recognition but that characteristic doesn't prohibit quantitative evaluation. The goal of any subjective evaluation should be to:

Minimize the probability that random chance accounts for the outcome

Maximize the agreement between raters in the experiment

These two goals are related. For example, suppose that ten people are rating dresses according to stylishness. You'd like to construct the evaluation such that the ten people are likely to agree on the level of stylishness. You'd also like to know that were you to select ten different people, you'd observe the same rating.

Ad-hoc evaluation of translation qualityi.e., sampling a few sentences and having translators count errorssatisfies neither of these goals.

This guide explains how to select an MT system in a principled and repeatable way. There are three approaches in increasing order of cost and time:

Automatic quality evaluation an algorithm determines or predicts the quality of the MT output. The most common use case is to compare the translation against a set of human-produced reference translations and calculate a score that reflects the quality. Automatic evaluation is cheap and fast.

Human quality evaluation experts (e.g. translators) look at the translated output, and give a score to the quality of the translation. Ratings can vary significantly across experts, so the evaluation should be designed to maximize agreement among raters. Human evaluation is costly and time-consuming.

Human productivity evaluation translators work with machine translation to produce final translations. The metrics are throughput (words per hour) and quality.

Start with an automatic evaluation to narrow the list of systems. Then use human evaluation to select the best system. Finally, conduct a human productivity evaluation to measure the impact of machine assistance on the translation workflow.

This article describes current best practices for human evaluation of translation quality. For the latest research on quality evaluation, see the annual reports from the Workshop on Machine Translation (WMT), the most recent of which was held in August of 2016.

Current approaches to translation quality evaluation use the method of pairwise comparison. Consider the evaluation of clothing:

EMNLP 2014 paper

One question that you might ask is:

How good does Jeremy look in a green suit?

Raters could evaluate Jeremy on a scale of 1-10 for some definition of "good." You could also ask:

Does Jeremy look better than, worse than, or the same in a green suit vs. a pink suit?

It has been found that humans tend to render more consistent judgments when comparing two items than when rating an item on an arbitrary scale. This finding goes for clothes, and for translation. In the WMT 2007 evaluation campaign, it was observed that inter-rater agreement was considerably higher for pairwise comparisons ("sentence ranking" in the table below) vs. evaluation of fluency/adequacy:

In this table, a value of K = 1.0 indicates perfect agreement among raters. Fluency / adequacy judgments, which were popular in the early 2000s, have been almost entirely abandoned in favor of pairwise comparisons.

Setup

Let's continue the example from the automatic quality evaluation. We have two test sets (emailand listing), each with source sentences and target references. We also have the output of the two systems B and C. Now we need:

Bilingual human raters the raters should be fluent in both languages, preferably with native proficiency in the target.

A ranking interface for collecting the human judgments.

For each source sentence, the ranking interface shows a target reference, and MT outputs with the system identity concealed. The ordering of the systems should be randomized across screens:

Ranking Interface

In our example, the interface would show only two system outputs: B and C. The pairwise comparison setup can be safely extended to a relative ranking of up to five system outputs.

It is common to have several human raters independently score each source sentence.

Analyzing the Results

For The ranking interface yields two kinds of data:

Relative ordinal ranks

Pairwise preferences B > C, B < C, C = B, etc.

Let's use the relative ordinal ranks to create a side-by-side comparison, which is used at Google and in other academic settings. We simply compute the average rank across all sentences in the test for each system (B and C). We can create a table like this:

System

email

listing

B

1.2

1.4

C

1.8

1.6

We can see that system B is superior for both test sets.

To recap, we used an automatic quality evaluation to eliminate three of the five systems under consideration. Then we ran a human quality evaluation to differentiate the final two systems. The human quality evaluation was constructed to maximize inter-rater agreement, so we have confidence that were we to re-run the evaluation with a different set of raters, we would observe the same outcome.

Advanced topic The pairwise preferences can be used for more sophisticated statistical analysis. To learn more, refer to section 4.3.2 of our .

This article describes a human translation productivity evaluation plan. The methodology is based on human-subjects experiments in the human-computer interaction (HCI) literature. For example experiments, see our case studies.

We assume that you will evaluate Lilt vs. at least one other translation tool that includes machine translation. But the evaluation plan can be used to compare other translation tools.

The goal of a productivity evaluation is to measure two variables:

Throughput source words translated per hour

Quality of the final translations

In a machine-assisted setting, these two variables are related. To maximize throughput, a translator could simply confirm the MT output for every source sentence. To correct for this bias, we typically multiply the raw throughput by the quality score for each translator. This yields quality-adjusted words per hour. For example, suppose that translator A translates at 800 words per hour with a quality score (from the human translation quality evaluation ) of 4.2. We compute:

800 * (4.2 / 5.0) = 672 quality-adjusted words per hour

Setup

Let's continue the example from the human translation quality evaluation. We have two test sets (emailand listing), each with source sentences and target references. We also have the output of the two systems B and C. Concretely, let's now assume that system B is Lilt and system C is a a public MT system integrated into another computer-aided translation tool. Now we need:

At least four translators An experiment with two translators is possible, but it will be harder to separate differences due to tooling vs. human performance.

A method for collecting timing data Lilt collects segment-level timing. Some other tools offer plugins to collect this data. In a proctored environment, you may time the translators.

Divide the evaluation into two timed sessions separated by an untimed break. Here we call those sessions "morning" and "afternoon". Randomize the pairings of systems/tools and data sets to mitigate the effects of fatigue, source text difficulty, and human proficiency with each tool. Here is an example design:

morning

afternoon

Translator 1

Lilt / email

Tool / listing

Translator 2

Tool / listing

Lilt / email

Translator 3

Lilt / listing

Tool / email

Translator 4

Tool / email

Lilt / listing

Analyzing the Results

Absent segment-level timings, simply aggregate the number of words translated and the total translation time for each system and test set. Create a table like this:

email WPH

listingWPH

Tool

431

529

Lilt

601

720

To compute quality-adjusted words per hour, repeat the human translation quality evaluation for the final translations produced by the human translators. Then scale the raw throughputs in the table above by the quality scores.

Advanced topic The segment-level timings can be used for more sophisticated statistical analysis. To learn more, refer to section 4.3.1 of our EMNLP 2014 paper.

Good Hygiene for Human Productivity Evaluation

Each subject should translate between 2,500 and 3,000 words per day. The industry average is 2,684 words per day. Translating more could increase the effect of fatigue.

Measure time accurately, either via software or by proctoring. Don't rely on the translators to time themselves.

Measure quality so that translators are incentivized to translate quickly and accurately.

Evaluate quality-adjusted throughput: throughput * quality.

Incentivize the participantsby giving a prize to the top performer on the evaluation metric. 1. The price could be as small as a free coffee. But there mustbe some incentive.

Limit time for self-review, which is a source of considerable variance among translators.