zulip/docs/processes/api-design.md

Designing changes to the Zulip API

One of Zulip's strengths is its clean, well-documented API which clients use to talk to the server. This HTTP-based API is used equally by the Zulip web/desktop app, the official mobile app and terminal app, and numerous third-party integrations, bots, and tools written by many different Zulip users.

Many new Zulip features involve making changes to the API. Because the Zulip API is so widely used, it's important that we do a good job of those changes, in several respects:

• All API changes must be compatible, meaning they don't cause existing Zulip clients (e.g., existing installs of the Zulip mobile app) to break.

• All API changes should be clean APIs, meaning the new API features have crisp meanings, will support clients implementing the feature in a natural way, and have clear names that point accurately to the meanings.

• All API changes should be clearly documented and follow our API documentation conventions.

The last point is covered in its own document. This document describes how we accomplish the other two: compatible API changes, and clean APIs.

API change process, in short

There are two essential rules for how we make API changes:

1. When considering an API change, always discuss it in the #api design channel.

2. Before merging a PR with an API change, always make sure the change has been approved in that channel.

As long as you follow those two rules, the maintainers who manage that channel can guide you through all the other aspects of the process, as described in this document.

(For what counts as an "API change", see the next section.)

What is an API change?

An "API change" is anything that adds, removes, or changes the meaning of anything that the server sends to clients, or that the server accepts from clients.

For example, this includes:

• adding or removing an endpoint (editing zproject/urls.py)
• adding or removing parameters of an endpoint
• adding or removing any fields in the initial snapshot (the /api/register-queue response) or any other endpoint's response
• changing the type of any of the above, including adding new values to an enum type
• renaming or moving any of the above
• changing the meaning of any existing parameter or field

In general, almost any change that touches the API spec file zerver/openapi/zulip.yaml will be an API change. (The few exceptions are things like clarifying the wording of existing documentation there.) Conversely, for most — but not all! — API changes, if you write tests that exercise the changes, then our test suite will ensure there's a corresponding change in the API spec file.

Some types of API changes that one might not realize are API changes:

• Any change to the structure of "Zulip content HTML", the HTML that the server emits for message content (and a few other places like channel descriptions). This includes:

  • new CSS classes
  • new data-* attributes or other attributes
  • existing CSS classes or attributes appearing in new places, or disappearing from some existing places

• Any change to push notification payloads.

All types of API changes need to follow the principles in this document and be discussed in #api design.

API design approval

The #api design channel has a few maintainers who have taken on the specific responsibility of managing that channel's discussions. Typically these include the lead of the Zulip mobile team, and one or two Zulip maintainers who work primarily in the server and/or web app.

A given API change proposal is approved when one of the channel's maintainers says, in the change's thread in the channel, that the change is approved.

Typically this will happen only after reaching consensus between both a mobile maintainer and a server or web maintainer, each agreeing that the change looks good from their perspective.

If one of the channel maintainers has said the change looks good and you're not sure if they mean only from their perspective (so are waiting for a perspective from a maintainer of the other system) or they mean the change is approved and ready to merge, please ask so they can make the status more explicit.

API compatibility

All changes to the Zulip API must be compatible: they must not cause existing Zulip clients to break.

For some kinds of API changes this is easy. For others it takes more work.

Changes that are automatically compatible

Some API changes are automatically compatible with no further effort.

A first question for any API change is to determine whether this happy circumstance applies, because it saves us from needing to do the API compatibility work described in the next section.

Some of the principles that maintainers will use in determining if a change is automatically compatible are:

• Adding new endpoints, or new parameters on an endpoint, is always compatible because existing clients won't interact with these.

  • This is true only so long as the meaning of existing requests doesn't change, though. For example, if a new parameter has a default which is different from the endpoint's old behavior, then that's a change in the meaning of the endpoint when the new parameter isn't sent, and therefore may not be compatible.

• Adding new fields to an endpoint's response, or a JSON object nested inside a response, is always compatible. A correct Zulip client already ignores any field it doesn't expect.

  • Conversely, this means that if an existing piece of API starts including by default data objects it previously didn't, that change is not automatically compatible, even if the new objects have some new field to distinguish them — because existing clients will ignore that field.

    For example, when we started including archived channels in the lists of channels in /api/register-queue, we didn't just start including them while adding an "is_archived": true flag to them, because if we did that then existing clients would have innocently treated those as additional non-archived channels (and therefore wrongly shown them in the list-of-channels screen, etc.). Instead more compatibility work was required, per the next section.

• In Zulip content HTML, adding new attributes is compatible. Adding new CSS classes, however, is not automatically compatible. In general, most changes to Zulip content HTML are not automatically compatible.

  Relevant discussion: #api design > HTML pattern for truly inline images @ 💬

• If a change affects only API features that are used only by the Zulip web app and no other clients, then the change is compatible even if it does things like remove or rename a field.

  Typically this is true if a feature is used only in the settings UI. It's sometimes true of other features too.

  These changes are compatible because the web app is deployed with the server, and clients automatically reload into the new web app version soon after a server upgrade. Managing such changes typically at most requires the Zulip Cloud team to split the web and server changes across two consecutive deployments, which does not justify permanent compatibility code.

  To determine if this is the case:

  • Check that the mobile app doesn't use the affected feature. Always confirm this explicitly before deciding the feature isn't used; occasionally the app does depend on an API feature one wouldn't have expected.

    This check needs to apply to both the latest version of the app and any previous version that is still supported. For example, if the mobile app previously used a given feature and then stopped doing so, we can't remove the feature until about 12 months after the mobile release that stopped using it. (The mobile team lead makes removal decisions based on data the app stores provide about the distribution of versions).

  • Quickly check if the latest zulip/zulip-terminal might use the feature, and file an issue if it might. It's typically a quick git grep. We do not block API changes on terminal app compatibility.

  • Consider whether it's likely that any third-party integrations, bots, tools, or other clients might use the feature. Typically we figure they'll definitely use certain key endpoints like /api/send-message; won't use admin-only settings; and things in between are less clear. When in doubt, assume a feature might be used. Zulip Cloud maintainers have access to logs that can help determine to what extent an API endpoint is currently used by third-party code in that environment, but we have no access to similar data for self-hosted environments.

Making a change compatible

When an API change isn't automatically compatible (as described in the preceding section), we have to do some more work to make it compatible.

The fundamental strategy here is to make the change opt-in. This means that existing clients, those unaware of the change, don't see the change.

The basic way to make a change opt-in is to have the affected endpoint take a new parameter that enables the change, defaulting to off. For example, allow_empty_topic_name on /api/get-messages

When an API change affects the content of events, we need to use a variation on that approach: we add one more flag to the client_capabilities parameter of /api/register-queue, again defaulting to off. The resulting event queue then applies the given "client capability", affecting all requests to /api/get-events that use that queue.

When a change is opt-in (through either of these ways or any other variation), it incurs an ongoing complexity cost in the server codebase because the server must continue to support the old form as well as the new form. For this reason it's good to eventually come back and remove support for the old form. You should add a TODO/compatibility marker in the server codebase detailing the plan for compatibility code that you expect to be able to delete when a given condition has occurred. You should review several examples of this pattern using git grep TODO/compat -A10 before writing such a marker.

That removal is itself an API change, though, so it needs to go through the process like any other change.

Clients can help reduce this maintenance cost for the server implementation by promptly upgrading to request and properly handle the new form of recently changed APIs. (Even if the client app team doesn't have time to fully implement the new feature that motivated the API change).

Clean APIs

Designing an API change cleanly isn't as immediately critical as making sure it's compatible: an API not being clean won't immediately cause anything to break.

It's valuable to the project, though, that most of our API meets a fairly high standard of cleanliness and clarity. (Not all of it; we've made some mistakes.)

This has helped us keep space for making more and more API changes without boxing ourselves in.

It's also an essential ingredient in enabling the developers of the Zulip mobile app to efficiently implement each new feature, without having to reverse-engineer the server's behavior or work around awkward quirks that complicate a client's implementation.

Similarly, one of Zulip's selling points for users is that our API lets users write custom tools that can do anything the official clients can do. For this promise to work out in practice, it's important that the API be reasonably clear and free of pitfalls.

Writing a clean API is a matter of taste, experience, and engineering judgement, and can't straightforwardly be codified. With that in mind, though, here are some broad principles and questions to ask when reviewing a proposed API change for cleanliness and clarity:

• Think through what a client-side implementation of the feature (e.g., for web, or for mobile) would look like. Are there API revisions that would make that simpler or clearer? Should logic for the API feature be implemented in the server, the client, or both?

• Think through the proposed semantics of the new API. Make sure they make sense and seem like they'll solve the intended problem.

• Think through the proposed data structures, and the names of parameters, fields, and enum values. Make sure they're clear, they match the semantics, and they aren't likely to accidentally suggest a meaning that's different from the actual meaning.

• Think about possible future extensions of the API feature that is being implemented. Could the API design being proposed force us to make an incompatible change if we want to add a natural extension of the current feature? (For example: Some early API fields were implemented as tuples, rather than objects, and all of those resulted in painful migration work to replace the tuples with objects as additional fields were desired).

API change process

The essential thing is that all API changes are discussed and approved in the #api design channel. See the description above.

More details on the process are below.

As a PR author

If you're the author of a PR, please go ahead and start the #api design thread yourself. Then put a link to it in the PR description.

For any discussion that isn't specifically related to the API changes, please use a different channel such as #backend. (You can cross-link, if a data model discussion has implications for the API, for example).

Keeping #api design focused is important for other participants such as the mobile team, so that they can be sure to follow the discussions on #api design that are about API changes and are important for them.

As a PR reviewer

If you're reviewing a PR and it seems to make changes to the API, check the PR description for a link to an #api design thread. If there isn't one already, ask about it in your review. You can also start the thread yourself.

If you're reviewing a PR and it's near ready to merge, check to see if the #api design thread has concluded with approval of a proposed change. Check also that the PR's API changes match the agreed-on plan. If the chat thread hasn't concluded, please post there to try to bring it toward a conclusion.

As an API reviewer

After an API change is approved on #api design, the author on the server side prepares or revises a PR to implement the change.

If you were involved in approving the API change — and particularly if maintaining our API quality is an important part of your work, e.g., because you're the lead of the mobile team — then it's helpful to review that PR for its API docs changes. That means totally ignoring the code and reading only api_docs/ and zerver/openapi/.

• Depending on the author's skills and their experience with Zulip development, it's common for the draft docs to be unclear or inaccurate. Feedback on those is very helpful.

• Sometimes the draft docs reveal a misunderstanding or miscommunication about the API semantics that were agreed to. Those cases are important to catch.

As a channel maintainer of #api design

If you're one of the maintainers responsible for the #api design channel, try to catch up on threads there each workday.

If a thread needs input from the system you work on (mobile, web, server), try to provide it or to identify someone to do so.

If the proposed changes look good from your perspective but still need feedback from another perspective (e.g., server/web vs. mobile), try to make that clear, so the PR's author or reviewer doesn't mistakenly think the changes are already approved. An example remark might be:

This looks good to me, but we need someone from the mobile team to confirm ...

Conversely, if all perspectives are either already represented or you believe they're unnecessary for the particular change, then say explicitly that the API changes are approved so that the PR's author and reviewer know they can move forward.

Once an API change is approved, consider filing issues to implement it for mobile and other clients. (Or asking, on the thread, for someone else to file those.) This is useful for high-priority changes in order to move quickly. For most changes, we wait until the server PR has been merged and a bot posts about it in #api documentation.

As the mobile team lead

If you're leading the mobile team, that probably means you're a maintainer of the #api design channel and an API reviewer for most API changes. See the corresponding subsections above.

After each API change is merged into the server, a bot will post a new thread in #api documentation. You should make sure you or someone else replies to explicitly decide if there should be a mobile issue to track making a corresponding change in the mobile app, and files or updates such an issue if needed. (See #api documentation > new feature level: 454 @ 💬.)