Syncing keys

A push of a project with 10,000 keys in which nothing changed is one small request. This works with hashes: you compute a hash per key locally, Tolk tells you which keys are new or changed, and you send only those.

Computing hashes

A key has two hashes, both SHA-1 in lowercase hexadecimal:

string

The identity of the key: SHA-1 of namespace + "\x1F" + key + "\x1F" + (context ?? ""). The separator is the unit separator U+001F. A key without a namespace has namespace * (like lang/{locale}.json). An empty context and no context give the same hash. With the hash, a key whose name is a whole sentence never ends up in a URL.

string | null

The source text: SHA-1 of the text. For a plural key, the SHA-1 of the JSON of the forms, sorted by category, with UTF-8 unescaped (like PHP json_encode with JSON_UNESCAPED_UNICODE). null without a source text.

function tolkKeyHash(string $namespace, string $key, ?string $context = null): string
{
    return sha1($namespace."\x1F".$key."\x1F".($context ?? ''));
}

/** @param array<string, string>|null $forms */
function tolkSourceHash(?string $value, ?array $forms = null): ?string
{
    if ($forms !== null && $forms !== []) {
        ksort($forms);

        return sha1(json_encode($forms, JSON_UNESCAPED_UNICODE | JSON_THROW_ON_ERROR));
    }

    return ($value === null || $value === '') ? null : sha1($value);
}

import { createHash } from 'node:crypto';

const sha1 = (text) => createHash('sha1').update(text, 'utf8').digest('hex');

export const tolkKeyHash = (namespace, key, context = null) =>
  sha1([namespace, key, context ?? ''].join('\x1F'));

export function tolkSourceHash(value, forms = null) {
  if (forms && Object.keys(forms).length > 0) {
    const sorted = Object.fromEntries(
      Object.keys(forms).sort().map((category) => [category, forms[category]]),
    );
    // Same as PHP json_encode: escape "/" and the line separators U+2028/U+2029.
    const json = JSON.stringify(sorted)
      .replace(/\//g, '\\/')
      .replace(/ /g, '\\u2028')
      .replace(/ /g, '\\u2029');
    return sha1(json);
  }
  return value ? sha1(value) : null;
}

Check your implementation with these examples:

Input	Hash
`hash` of namespace `*`, key `Pay now`, no context	`0e5e60b24eff16607ccbb7dbbe7141b7a0c5de48`
`hash` of namespace `checkout`, key `Order`, context `noun`	`6a769493d0f894ecd2d4bb324ac5113a61056602`
`sourceHash` of `Nu betalen`	`09269d72c5af31fb3cc4752832089843eaaeb603`
`sourceHash` of `{"one": "Eén artikel", "other": ":count artikelen"}`	`8d7fabf1eb21b47eac0cf2d9a47e340abf7a51b8`

A push in two steps

Compare: POST /keys/diff

Send the hash and sourceHash of every local key (at most 10,000). Tolk responds with which hashes are new, which are changed (a different source text) and which are restored (archived in Tolk), and how many are unchanged. The diff changes nothing.

curl https://api.tolk.every-day.io/v1/keys/diff \
  -H "Authorization: Bearer $TOLK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keys": [{"hash": "0e5e60b24eff16607ccbb7dbbe7141b7a0c5de48", "sourceHash": "09269d72c5af31fb3cc4752832089843eaaeb603"}]}'

Send: POST /keys/sync

Send only the new, changed and restored keys, with their source text (at most 5,000 per request). A sync is safe to repeat: after a 409, that is how you finish the job.

{
  "keys": [
    {
      "namespace": "*",
      "key": "Pay now",
      "value": "Nu betalen",
      "description": "Button on the checkout page",
      "maxLength": 20,
      "tags": ["checkout"]
    },
    {
      "namespace": "cart",
      "key": "items",
      "forms": { "one": "Eén artikel", "other": ":count artikelen" }
    }
  ]
}

The response counts created, updated, restored, unchanged and archived, and lists in rejected the keys that Tolk refused, with the reason (for example plural_changed: an existing key switches between singular and plural).

Clean up (optional)

To archive keys that have disappeared from the code, page through GET /keys, compare the hashes with the local ones and send the missing hashes in the archive field of a sync. Tolk never archives on its own.

After a sync with new or changed keys, Tolk sends the keys.synced webhook.

Key limit

The customer’s plan limits the number of active keys across all of its projects. Tolk checks a sync up front as a whole: new keys plus restored archived keys, minus the keys the same sync archives. A sync that archives as many keys as it adds therefore always fits. If it does not fit, Tolk writes nothing and responds with a 422 (the message follows Accept-Language; here en):

{
  "message": "This customer has reached its limit of 5,000 keys: …",
  "errors": { "keys": ["This customer has reached its limit of 5,000 keys: …"] },
  "reason": "key_limit_reached",
  "limit": 5000,
  "used": 4980,
  "requested": 40
}

Do not repeat such a request: first archive keys that are no longer used, or ask Every Day for a larger plan. POST /keys/diff and reading keep working.

Reading and updating keys and translations

Endpoint	Purpose
`GET /keys`	Keys with filters (`namespace`, `tag`, `archived`, `updatedSince`) and pagination
`PATCH /keys/{keyHash}`	Change the description, maximum length and tags of a key
`GET /translations/{locale}`	Translations of a language, with filters (`status`, `namespace`, `tag`, `updatedSince`)
`PUT /translations/{locale}`	Update up to 1,000 translations at once, with `mode` `overwrite` or `fill_empty`
`GET /stats`	The progress per language

The fields and responses are in the API reference.

​Computing hashes

​A push in two steps

​Key limit

​Reading and updating keys and translations

Computing hashes

A push in two steps

Key limit

Reading and updating keys and translations