The searches we use to check search on the seed data, and again every time it is tweaked. The data is data/recommended.json (8 catalogs) and data/libraries.json (Bibliothèque numérique de Paris, Lirtuel, De Openbare bibliotheek Vlaanderen en Brussel, Médiathèque Valais). In the tables, BnParis is Bibliothèque numérique de Paris, De Openbare is De Openbare bibliotheek Vlaanderen en Brussel, BN Romande is Bibliothèque numérique Romande.
Expected is the answer we want. Today is what search returns, pinned exactly by the tests: word matches first, then catalogs found only through trigrams. Score is how close Today is to Expected. A row with a note is worth a look; it is not a bug in the data.
Place names are loaded in English plus the official languages of the country (French for France; French, Dutch and German for Belgium; French, German and Italian for Switzerland).
make search-score # score the running server (http://localhost:8000)
make search-score ARGS='--url https://… -v' # a deployed instance, with every row
make test # the same searches as exact-result tests
tests/integration/test_search_table.pytests/e2e/test_search_score.pytests/search_quality.py, scripts/score_search.pytests/search_cases.pyperformance.mdWith today’s results the score is 0.877 over 90 scored searches; 73 are exactly the ideal answer. Mean recall 0.95, mean precision 0.88 (searches whose ideal is nothing are not in those two).
A search scores 1.0 when it returns the expected catalogs, in the expected order, and nothing else. It is ndcg x precision: recall is how many expected catalogs came back, precision is how many of the catalogs returned were expected (noise lowers it), and ndcg rewards the right order (expected catalogs counting more the earlier they appear). A search whose ideal is nothing scores 1.0 for nothing and 0.0 for anything. The headline number is the mean over all scored searches, so a tweak to ranking, weights or the typo threshold becomes one number to compare.
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
Paris |
Title word, and since the city was filled in also the city | BnParis | BnParis | 1.00 | The city (Paris, label C) adds to the title match: the score went from 0.608 to 0.638. |
ile-de-france |
Region name with hyphens | BnParis | BnParis | 1.00 | |
Île de France |
Same name typed without hyphens | BnParis first | BnParis, De Openbare, Lirtuel | 0.33 | Both Belgian libraries come back because of ‘de’ (Région de Bruxelles-Capitale, De Openbare). Accepted for now (Q5): there are no stop words. |
France |
Country name, weight B | BnParis | BnParis | 1.00 | |
Belgique |
Country, French | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
Belgium |
Country, English | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
België |
Country, Dutch, accent folded | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
Belgien |
Country, German | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
Bruxelles |
Region, French | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
Brussels |
Region, English | Lirtuel, De Openbare | Lirtuel, De Openbare | 1.00 | |
Brussel |
Region, Dutch, also in a title | De Openbare, Lirtuel (De Openbare first) | De Openbare, Lirtuel | 1.00 | |
Brüssel |
Region, German | De Openbare, Lirtuel (De Openbare first) | De Openbare, Lirtuel | 1.00 | |
Wallonie |
Region, French | Lirtuel | Lirtuel | 1.00 | |
Wallonia |
Region, English | Lirtuel | Lirtuel | 1.00 | |
Vlaanderen |
Region, Dutch everyday name, also in the title | De Openbare | De Openbare | 1.00 | |
Flandre |
Region, French everyday name added by hand | De Openbare | De Openbare | 1.00 | |
Flanders |
Region, English | De Openbare | De Openbare | 1.00 | |
Valais |
Canton, French and English, also in the title | Médiathèque Valais | Médiathèque Valais | 1.00 | |
Wallis |
Canton, German | Médiathèque Valais | Médiathèque Valais, Lirtuel (trigram only) | 0.50 | Lirtuel follows as a trigram match (Wallonia is close to Wallis) since the typo threshold moved to 0.5. |
Vallese |
Canton, Italian | Médiathèque Valais | Médiathèque Valais | 1.00 | |
Suisse |
Country, French | Médiathèque Valais | Médiathèque Valais | 1.00 | |
Schweiz |
Country, German | Médiathèque Valais | Médiathèque Valais | 1.00 | |
Svizzera |
Country, Italian | Médiathèque Valais | Médiathèque Valais | 1.00 | |
Bristol |
Place with no catalog, close to Brussel | nothing | nothing | 1.00 | Trigrams don’t pull in the Belgian libraries. |
région |
Word found in Belgian region names | Lirtuel, De Openbare | De Openbare, Lirtuel | 1.00 | |
canton |
Word found in Swiss canton names | Médiathèque Valais | Médiathèque Valais | 1.00 |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
Belgio |
Italian name of Belgium, Italian isn’t official there | nothing | Lirtuel (trigram only), De Openbare (trigram only) | 0.00 | Found through trigrams, close to Belgie and Belgien. Harmless, arguably helpful. |
Vallonia |
Italian name of Wallonia | nothing | Lirtuel (trigram only) | 0.00 | Found through trigrams, close to Wallonia. Same as above. |
Zwitserland |
Dutch name of Switzerland, Dutch isn’t official there | nothing | nothing | 1.00 |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
Gutenberg |
Title word | Project Gutenberg | Project Gutenberg | 1.00 | |
bibliotheque |
Title word without accents | BnParis, BN Romande, Bibliothèque russe et slave | BnParis, BN Romande, Bibliothèque russe et slave, De Openbare (trigram only) | 0.75 | De Openbare bibliotheek also comes back through trigrams, after the three exact matches. |
MÉDIATHÈQUE |
Upper case with accents | Médiathèque Valais | Médiathèque Valais | 1.00 | |
ebooks |
Shared title word | Standard Ebooks, Ebooks libres et gratuits | Standard Ebooks, Ebooks libres et gratuits | 1.00 | |
standard ebooks |
Two words, any-word matching | Standard Ebooks, Ebooks libres et gratuits (Standard Ebooks first) | Standard Ebooks, Ebooks libres et gratuits | 1.00 | |
Liber |
Short title word, close to libres | Liber Liber | Liber Liber, Librivox (trigram only), Ebooks libres et gratuits (trigram only) | 0.33 | Librivox and Ebooks libres follow as trigram matches since the typo threshold moved to 0.5. |
Librivox |
Title word | Librivox | Librivox | 1.00 | |
Bibliothèque Nationale |
Two words, only one of them in a title | BnParis, BN Romande, Bibliothèque russe et slave | BnParis, BN Romande, Bibliothèque russe et slave | 1.00 | There’s no national library in the seed data, so these match on ‘bibliothèque’ alone and have the same score. |
TV5 |
Word with a digit | TV5 Monde | TV5 Monde | 1.00 |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
gutenbrg |
Missing letter in a title | Project Gutenberg | Project Gutenberg (trigram only) | 1.00 | |
bruxels |
Missing letters in a place name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
belgiqe |
Missing letter in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
bibliothèques |
Plural | BnParis, BN Romande, Bibliothèque russe et slave | BnParis (trigram only), BN Romande (trigram only), Bibliothèque russe et slave (trigram only), De Openbare (trigram only) | 0.75 | |
guten |
Start of a word | Project Gutenberg | Project Gutenberg (trigram only) | 1.00 | |
parsi |
Two letters swapped in a short word | BnParis | BnParis (trigram only) | 1.00 | Found through trigrams since the typo threshold moved to 0.5 (nothing at 0.6). Swapped letters stay the weakest typo: about two in three are found. |
Suiße |
ß is folded by the database’s unaccent, not by Python | Médiathèque Valais | Médiathèque Valais | 1.00 |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
belgque |
One letter dropped in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
bruxeles |
One letter dropped in a place name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
libriox |
One letter dropped in a title | Librivox | Librivox (trigram only), Ebooks libres et gratuits (trigram only) | 0.50 | |
valas |
One letter dropped in a canton | Médiathèque Valais | Médiathèque Valais (trigram only) | 1.00 | |
walonie |
One letter dropped in a region | Lirtuel | Lirtuel (trigram only) | 1.00 | |
flandrs |
One letter dropped in a region | De Openbare | De Openbare (trigram only) | 1.00 | |
suise |
One letter dropped in a country name | Médiathèque Valais | Médiathèque Valais (trigram only) | 1.00 | |
standrd |
One letter dropped in a title word | Standard Ebooks | Standard Ebooks (trigram only) | 1.00 | |
pairs |
Two neighbouring letters swapped in a title word | BnParis | nothing | 0.00 | Not found: swapped or replaced letters in a short word share too few trigrams. |
valias |
Two neighbouring letters swapped in a canton | Médiathèque Valais | nothing | 0.00 | Not found: swapped or replaced letters in a short word share too few trigrams. |
brussles |
Two neighbouring letters swapped in a place name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
gutneberg |
Two neighbouring letters swapped in a title | Project Gutenberg | nothing | 0.00 | Not found: swapped or replaced letters in a short word share too few trigrams. |
belguim |
Two neighbouring letters swapped in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
wallonei |
Two neighbouring letters swapped in a region | Lirtuel | Lirtuel (trigram only) | 1.00 | |
schwiez |
Two neighbouring letters swapped in a country name | Médiathèque Valais | Médiathèque Valais (trigram only) | 1.00 | |
bibliotehque |
Two neighbouring letters swapped in a title word | BnParis, BN Romande, Bibliothèque russe et slave | BnParis (trigram only), De Openbare (trigram only), BN Romande (trigram only), Bibliothèque russe et slave (trigram only) | 0.68 | |
parus |
One letter replaced by a neighbouring key in a title word | BnParis | BnParis (trigram only) | 1.00 | |
belgiqie |
One letter replaced by a neighbouring key in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
vqlais |
One letter replaced by a neighbouring key in a canton | Médiathèque Valais | nothing | 0.00 | Not found: swapped or replaced letters in a short word share too few trigrams. |
frabce |
One letter replaced by a neighbouring key in a country name | BnParis | BnParis (trigram only) | 1.00 | |
lirtuek |
One letter replaced by a neighbouring key in a title | Lirtuel | Lirtuel (trigram only) | 1.00 | |
bibliotheuqe |
One letter replaced by a neighbouring key in a title word | BnParis, BN Romande, Bibliothèque russe et slave | BnParis (trigram only), De Openbare (trigram only), BN Romande (trigram only), Bibliothèque russe et slave (trigram only) | 0.68 | |
guttenberg |
One letter doubled or added in a title | Project Gutenberg | Project Gutenberg (trigram only) | 1.00 | |
belgiumm |
One letter doubled or added in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
bruxellles |
One letter doubled or added in a place name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 | |
pariss |
One letter doubled or added in a title word | BnParis | BnParis (trigram only) | 1.00 | |
librivoxx |
One letter doubled or added in a title | Librivox | Librivox (trigram only) | 1.00 | |
standrad ebooks |
Two words, a swap in one of them | Standard Ebooks, Ebooks libres et gratuits (Standard Ebooks first) | Standard Ebooks, Ebooks libres et gratuits | 1.00 | |
belgqiue |
Two slips in one word in a country name | Lirtuel, De Openbare | Lirtuel (trigram only), De Openbare (trigram only) | 1.00 |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
"numérique de paris" |
Quoted phrase | BnParis | BnParis, BN Romande (trigram only) | 0.50 | Romande follows as a trigram match: the fuzzy half ignores the quotes. |
bibliothèque -paris |
Excluding a word | BN Romande, Bibliothèque russe et slave | BN Romande, Bibliothèque russe et slave, De Openbare (trigram only) | 0.67 | Negated words exclude catalogs from both halves, so BnParis never comes back. |
Belgique -lirtuel |
Negation keeps a real score | De Openbare | De Openbare | 1.00 | |
paris OR valais |
OR | BnParis, Médiathèque Valais | Médiathèque Valais, BnParis | 1.00 | |
PARIS or |
A bare OR is dropped | BnParis | BnParis | 1.00 | |
"de paris |
Unbalanced quote | no error | BnParis, De Openbare, Lirtuel | — | No error is the requirement. Treated as the words ‘de’ and ‘paris’; ‘de’ noise as above. |
brussel - |
Lone dash | De Openbare, Lirtuel (De Openbare first) | De Openbare, Lirtuel | 1.00 | As Brussel, no error. |
& \| ! : |
Special characters only | nothing | nothing | 1.00 | Nothing, and no error. |
xyzzy |
No match at all | nothing | nothing | 1.00 | |
| (empty) | Empty search | nothing | nothing | 1.00 | Decided: stays empty, no database work. |
| (spaces only) | Search with only spaces | nothing | nothing | 1.00 | Same as empty. |
-paris |
Only a negation | nothing | nothing | 1.00 | Same as empty. |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
bibliothèque de Paris |
Three words, one of them ‘de’ | BnParis first | BnParis, De Openbare, BN Romande, Bibliothèque russe et slave, Lirtuel | 0.20 | De Openbare comes second because of ‘de’ in its title. The stop word problem (Q5). |
| Search | What it checks | Expected | Today | Score | Note |
|---|---|---|---|---|---|
Italy |
Country of a catalog with no country set (Liber Liber) | nothing | nothing | 1.00 | Correct for the data we have. |
Italia |
Same, in Italian | nothing | nothing | 1.00 | |
Italie |
Same, in French | nothing | nothing | 1.00 | |
United States |
Same, for Project Gutenberg and Standard Ebooks | nothing | nothing | 1.00 |
The real page size is 50. Twelve catalogs cannot fill a page, so the same query is read with a smaller page. Each page is listed between the slashes.
| Search | What it checks | Expected | Today | Note |
|---|---|---|---|---|
bibliothèque de Paris, 2 per page, pages 1 to 3 |
Pages add up to the full list, nothing repeated or skipped | BnParis, De Openbare / BN Romande, Bibliothèque russe et slave / Lirtuel | BnParis, De Openbare / BN Romande, Bibliothèque russe et slave / Lirtuel | (empty) |
Belgique, 1 per page, pages 1 to 2 |
Two catalogs with the same score keep the same order, run after run | Lirtuel / De Openbare | Lirtuel / De Openbare | Both have exactly the same score. The order comes from the tie-breaker (import date, then id), which the query has to keep. |
More paging checks (page past the end, links, the 1,000 page limit, 120 catalogs over three pages) are in tests/e2e/test_search_pagination.py and tests/integration/test_search_table.py.
Synthetic catalogs stand in for libraries that do not exist yet: one covering New Caledonia (FR-NC), one Guyane (FR-973), both with country FR. Nouvelle-Calédonie, New Caledonia, nouvelle caledonie, Guyane, French Guiana and the typo guyanne find them, and France finds the Paris library and both. See tests/integration/test_search_table.py.
A city goes into the search with the same weight as a subdivision (label C), below the country. The Paris library now carries city: Paris, but its title also says Paris, so no seed search can show the city alone. That is tested with synthetic catalogs: a city-only match is found, a title match outranks a country match which outranks a city match, and changing or removing the city re-indexes the row. See tests/integration/test_search_table.py and tests/integration/test_search_lifecycle.py.
Place names work in every language we load, with and without accents, and typos in both titles and place names are found; plurals come through trigrams. Things to watch: short common words match because there are no stop words yet, so ‘de’ pulls in the Belgian libraries; swapped letters in short words are found only about two times in three; and a quoted phrase gets fuzzy extras after the exact match.