catalog-registry

`make add` importing a catalog from its feed

One command imports one catalog from a live OPDS 2.0 endpoint, straight into Postgres.

make add ARGS="https://example.org/opds --kind public"

created: Example Library

Run it again and it prints updated:, the import is idempotent.

make seed is the other way in, importing data/recommended.json. Both paths share the same validation and the same upsert. See development.md.

What it takes from the feed, and what you supply

The feed answers what this catalog is called and where its parts live. It cannot answer what kind of institution this is, no OPDS document expresses that, so the rest is flags.

	Source
`title`	the feed’s `metadata.title`
`links`	the feed’s `links`
`kind`	`--kind`, the only required flag
everything else	optional flags, below

--kind              open | public | academic | school | specialized   (repeatable, required)
--color             gray | red | yellow | blue | green | purple | orange | pink
--description       free text
--language          BCP-47, repeatable        --country      ISO 3166-1 alpha-2, uppercase
--subdivision       ISO 3166-2, upper, repeat* --city        free text
--coverage          global | country | subdivisions | local
--publication-type  ebook | audiobook | comic | newspaper | magazine | journal | article
--dry-run           print the document, write nothing

Two behaviours worth knowing before you fill these in:

--language changes who sees the catalog. A catalog declaring no language appears for every Accept-Language. Declaring en hides it from a reader asking for French.
Omitting --coverage stores NULL, not global., the registry does not assert worldwide reach on a catalog’s behalf.

What it does with the links

The URL you typed becomes the catalog link. The rest come from the feed, mapped onto the registry’s eight rels:

A link’s rel may be a string or an array, and an array may name relations this registry does not store. Those are skipped, and the rest are kept: ["catalog", "start"] stores one catalog link.

In the feed	Stored as
the URL you typed	`catalog`
`shelf`, or `http://opds-spec.org/shelf`	`shelf`
`search`	`search`, `templated` preserved
`profile`, `icon`, `alternate`, `authenticate`	unchanged
anything else, `self` and paging rels included	dropped

* --subdivision only accepts codes already in the subdivisions table. That table is not a full copy of ISO 3166-2. It holds the codes some catalog already uses, five at the time of writing. A valid code that is missing is refused with a message telling you to add it in a migration; see migrations/versions/b41f7c9ade52_… for the shape.

Your URL becomes catalog, never the feed’s own self. A remote feed’s self is not something this registry trusts for anything.

For a catalog add created, the identity later imports match on is that catalog link. Re-running add on the same URL finds the row it created the first time and updates it. No identifier is generated: the input schema does not require one, and the import derives catalogs.id as uuid5(NAMESPACE_URL, <that URL>). So the id is a function of the URL you typed, the same URL gives the same id on any machine, and --dry-run twice produces byte-identical documents. Change the URL and you get a second catalog, not an updated one.

The href is the fallback, not the only rule. The import matches on metadata.identifier first when a document supplies one, and on the catalog/shelf href otherwise. add never supplies one, so it always takes the href branch. Every entry in data/recommended.json does, which is why a catalog in that file survives a feed-URL change as one updated row while an add-created one becomes a second row. Adding an identifier is how you opt into that.

http://opds-spec.org/shelf is the single alias accepted. OPDS 1.2 §6.1 defines the relation that way and gives it no short form. There is no generic prefix rule, so a rel that the specifications define but this registry does not store is dropped like any other, and silently. That covers http://opds-spec.org/subscriptions, facet and crawlable. Use --dry-run to see exactly which links an import would keep.

Re-running replaces the whole record

Not a patch. Child collections are deleted and rebuilt, and every omitted flag reverts to its default:

make add ARGS="<url> --kind public --color purple"   # colour purple
make add ARGS="<url> --kind public"                  # colour is gray again

Correct for an import - the feed plus your flags are the whole truth for that catalog - and wrong for editing one field. Partial edits are the back office’s job.

Two things it does not set

status is always active and recommended is always true. There is no flag for either, so anything you add is in the public feed immediately. Marking a catalog recommended, or not, is Phase 3.

When it refuses

Situation	Message
not `http`/`https`	`... is not an http(s) URL`
not a public address	`... resolves to 10.0.0.5, which is not a public address`
redirected off http(s), or to a private address	`refusing to follow a redirect to ...`
unreachable, or times out after 12s	`could not fetch ..., <reason>`
over 2 MB	`... returned more than 2097152 bytes`
not JSON, or not a JSON object	`... did not return JSON`
no `metadata.title`	`... has no metadata.title, so there is nothing to name it`
no `catalog` or `shelf` link	`... needs a catalog or shelf link`
a field the schema rejects	the validation error, with its path

Nothing is written unless the whole document validates.

The address check is not only a safety measure. A catalog in a public registry has to be reachable by readers, so localhost, 10.x, 192.168.x and 169.254.x are all refused, and every redirect hop is checked again because a public host is free to redirect to a private one. That does mean you cannot import from a feed served on your own machine.

Search updates itself

A catalog’s search row is rebuilt by database triggers in the same transaction as the import, so make add, make seed and make seed-libraries need no extra step and a search finds the new or changed catalog straight away, including a changed city. After a bulk import of many catalogs run VACUUM ANALYZE catalog_search;. See search.md.

Where it writes

Directly to Postgres, using REGISTRY_DATABASE_URL. It is a database client, like psql, it never calls the API, which is why make logs stays silent during an import, and why it needs make up running only for the database, not the service.

It is also as privileged as the DSN it holds. With the Cloud SQL Auth Proxy up, that DSN can be production.