CodeCow/zulip
1
0
Fork 0
mirror of https://github.com/zulip/zulip.git synced 2026-07-03 21:10:12 +08:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
ccfcb34bea
zulip/zerver/lib
History
Alex Vandiver 07c4291749 message: Rewrite personals query to be more performant and accurate. 
The previous query suffered from bad corner cases when the user had
received a large number of direct messages but sent very few,
comparatively.  This mean that the first half of the UNION would
retrieve a very large number of UserMessage rows, requiring fetching a
large number of Message rows, merely to throw them away upon
determining that the recipient was the current user.

Instead of merging two queries of "last 1k received" + "last 1k sent",
we instead make better use of the UserMessage rows to find "last 1k
sent or received."  This may change the list of recipients, as large
disparities in sent/received messages may result in pushing the
most-recently-sent users off of the list.  These are likely uncommon
edge cases, however -- and the disparity is the whole reason for the
performance problem.

This also provides more correct answers.  In the case where a user's
1001'th message sent was to person A today, but my most recent message
received was from them yesterday, the previous plan would show the
message I received yesterday message-id as the max, and not the more
recent message I sent today.

While we could theoretically raise the `RECENT_CONVERSATIONS_LIMIT` to
more frequently match the same recipient list as previously, this
increases the cost of the most common cases unreasonably.  With a
1000-message limit, the common cases are slightly faster, and the tail
latencies are very much improved; raising `RECENT_CONVERSATIONS_LIMIT`
would increase the result similarity to the old algorithm, at the cost
of the p50 and p75.

|        |   Old   |   New   |
| ------ | ------- | ------- |
| Mean   | 0.05287 | 0.02520 |
| p50    | 0.00695 | 0.00556 |
| p75    | 0.05592 | 0.03351 |
| p90    | 0.14645 | 0.08026 |
| p95    | 0.20181 | 0.10906 |
| p99    | 0.30691 | 0.16014 |
| p99.9  | 0.57894 | 0.19521 |
| max    | 22.0610 | 0.22184 |

On the whole, however, the much more bounded worst case are worth the
small changes to the resultset.
2024-01-18 09:30:20 -08:00
..
markdown markdown: Prevent OverflowError with large time integers. 2024-01-05 12:01:06 -08:00
upload s3: Add a setting for S3 addressing style. 2024-01-05 11:12:18 -08:00
url_preview python: Use urlsplit instead of urlparse. 2023-12-05 13:03:07 -08:00
webhooks mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
__init__.py
addressee.py actions: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
alert_words.py models: Extract zerver.models.alert_words. 2023-12-16 22:08:44 -08:00
async_utils.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
attachments.py models: Move some functions to zerver.lib.attachments. 2023-12-16 22:08:44 -08:00
avatar_hash.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
avatar.py python: Consistently use from…import for urllib.parse. 2023-12-05 13:03:07 -08:00
bot_config.py ruff: Fix UP032 Use f-string instead of format call. 2023-08-02 15:58:55 -07:00
bot_lib.py models: Extract zerver.models.users. 2023-12-16 22:08:44 -08:00
bot_storage.py python: Import F, Q, QuerySet from their canonical module. 2023-03-05 14:46:28 -08:00
bulk_create.py models: Extract zerver.models.groups. 2023-12-16 22:08:44 -08:00
cache_helpers.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
cache.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
camo.py typing: Apply trivial none-checks with assertions as necessary. 2022-06-23 19:25:48 -07:00
ccache.py python: Delete superfluous parens. 2023-09-13 13:40:19 -07:00
compatibility.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
context_managers.py
create_user.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
data_types.py python: Elide unnecessary list wrappers. 2023-09-13 12:41:23 -07:00
db.py test_helpers: Fix logging in cursor_executemany mock. 2023-11-15 15:27:54 -08:00
debug.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
default_streams.py streams: Pass stream_weekly_traffic field in stream objects. 2023-08-06 18:06:42 -07:00
dev_ldap_directory.py dev_ldap_directory: Use f-strings for better readability. 2024-01-09 12:09:09 -08:00
digest.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
display_recipient.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
domains.py
drafts.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_mirror_helpers.py email_mirror: Move ZulipEmailForwardUserError into email_mirror_helpers. 2021-08-31 16:37:54 -07:00
email_mirror.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_notifications.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_validation.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
emoji_utils.py emoji: Match emoji sequences in markdown. 2023-08-23 16:18:15 -07:00
emoji.py models: Extract zerver.models.realm_emoji. 2023-12-16 22:08:44 -08:00
event_schema.py events: Add 'onboarding_steps' event deprecating 'hotspots'. 2023-12-06 18:19:20 -08:00
events.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
exceptions.py remote_billing_page: Show error page for registration mismatch. 2023-12-10 19:33:48 -08:00
export.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
external_accounts.py ruff: Fix SIM118 Use k not in d instead of k not in d.keys(). 2023-07-24 10:39:28 -07:00
fix_unreads.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
generate_test_data.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
github.py ruff: Fix RSE102 Unnecessary parentheses on raised exception. 2023-02-04 16:34:55 -08:00
home.py views: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
hotspots.py banner: Make banner about automatic follow/unmute topics one-time only. 2023-12-06 18:19:20 -08:00
html_diff.py html_diff: Fix lxml import. 2023-03-05 14:46:28 -08:00
html_to_text.py widgets: Rename confusing attribute name in tabbed_sections.py. 2023-08-31 11:55:28 -07:00
i18n.py i18n: Fix default language for users created via API/LDAP. 2023-10-01 21:10:13 +02:00
import_realm.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
initial_password.py initial_password: Add explicit development environment assertion. 2022-03-21 12:05:59 -07:00
integrations.py integrations: Extract integration event types returning function. 2023-08-30 15:54:13 -07:00
logging_util.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
management.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
mdiff.py node_tests: Move to web/tests. 2023-02-23 16:04:17 -08:00
mention.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
message.py message: Rewrite personals query to be more performant and accurate. 2024-01-18 09:30:20 -08:00
migrate.py migration: Make 'rename_indexes_constraints' a lib function. 2023-12-06 18:19:20 -08:00
mobile_auth_otp.py
muted_users.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
name_restrictions.py requests: Add SELF_HOSTING_MANAGEMENT_SUBDOMAIN. 2023-11-22 14:22:26 -08:00
narrow_helpers.py narrow: Split out narrow_helpers. 2023-06-30 11:26:23 -07:00
narrow.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
notes.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
notification_data.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
onboarding.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
outgoing_http.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
outgoing_webhook.py models: Extract zerver.models.bots. 2023-12-16 22:08:44 -08:00
per_request_cache.py per-request caches: Add per_request_cache library. 2023-08-11 11:09:34 -07:00
presence.py models: Move query_for_ids to zerver.lib.query_helpers. 2023-12-16 22:08:44 -08:00
profile.py profile: Strengthen decorator types using ParamSpec. 2022-04-14 12:44:35 -07:00
push_notifications.py zilencer: Have server send realm_uuid to remaining bouncer endpoints. 2024-01-05 13:09:09 -08:00
pysa.py
query_helpers.py models: Move query_for_ids to zerver.lib.query_helpers. 2023-12-16 22:08:44 -08:00
queue.py queue: Only NAK the events if the channel is still open. 2023-12-12 09:20:29 -08:00
rate_limiter.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
realm_description.py markdown: Refactor out additional properties added to Message. 2021-06-24 18:14:53 -07:00
realm_icon.py settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
realm_logo.py settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
recipient_parsing.py request: Extract out methods from 'scheduled_messages' to reuse. 2023-10-10 17:15:28 -07:00
recipient_users.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
redis_utils.py
remote_server.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
request.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
response.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
rest.py ruff: Fix RSE102 Unnecessary parentheses on raised exception. 2023-02-04 16:34:55 -08:00
retention.py models: Always search Messages with a realm_id or id limit. 2023-09-11 15:00:37 -07:00
safe_session_cached_db.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
scheduled_messages.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
scim_filter.py scim: Order Users by id when queried using filter syntax. 2021-11-26 16:06:16 -08:00
scim.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
send_email.py send_email: Distinct emails means distinct, case-insensitively. 2024-01-04 10:46:53 -08:00
server_initialization.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
sessions.py models: Extract zerver.models.users. 2023-12-16 22:08:44 -08:00
singleton_bmemcached.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
soft_deactivation.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
sounds.py actions: Split out zerver.lib.sounds. 2022-04-14 14:26:40 -07:00
sqlalchemy_utils.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
storage.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
stream_color.py streams: Extract stream_color library. 2022-03-14 18:01:36 -07:00
stream_subscription.py streams: Send user creation events on adding subscribers. 2023-11-21 23:58:45 -08:00
stream_topic.py message_send: Handle notifications for UNMUTED topic in a muted stream. 2023-03-06 19:15:45 -08:00
stream_traffic.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
streams.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
string_validation.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
subdomains.py python: Consistently use from…import for urllib.parse. 2023-12-05 13:03:07 -08:00
subscription_info.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
templates.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
test_classes.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
test_console_output.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
test_data.source.txt Rename default branch to ‘main’. 2021-09-06 12:56:35 -07:00
test_fixtures.py ruff: Fix ANN204 missing return type annotation for __init__. 2022-11-16 09:29:11 -08:00
test_helpers.py auth: Add a configurable wrapper around authenticate calls. 2024-01-15 12:18:48 -08:00
test_runner.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
tex.py push_notification: Fix bad rendering of math formulas. 2023-11-26 23:30:24 -08:00
thumbnail.py docs: Remove some outdated references to thumbnailing.md doc. 2022-07-12 17:44:24 -07:00
timeout.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
timestamp.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
timezone.py timezone: Improve tzdata parser’s compatibility with zic(8). 2022-09-20 16:58:31 -07:00
topic.py topic: Add comments calling out case-sensitive index usage. 2023-09-27 10:22:42 -07:00
transfer.py uploads: Allow uploads to set storage class. 2023-07-19 16:19:34 -07:00
typed_endpoint.py remote_realm: Add syncing of org_type. 2023-11-28 14:41:16 -08:00
types.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
url_encoding.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
url_redirects.py help: Rename view-and-browse-images.md -> view-images-and-videos.md 2023-11-23 10:37:31 -08:00
user_agent.py
user_counts.py python: Elide unnecessary list wrappers. 2023-09-13 12:41:23 -07:00
user_groups.py models: Extract zerver.models.groups. 2023-12-16 22:08:44 -08:00
user_message.py actions: Split out zerver.lib.user_message. 2022-04-14 17:14:30 -07:00
user_status.py users: Update presence and user status code to support restricted users. 2023-11-21 23:58:45 -08:00
user_topics.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
users.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
utils.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
validator.py realm: Enfore length restriction on jitsi_server_url at API level. 2023-12-14 12:11:59 -08:00
widget.py widgets: Fix bug where a new line right after /todo broke rendering. 2023-09-08 15:39:07 -07:00
zcommand.py ruff: Fix UP032 Use f-string instead of format call. 2023-08-02 15:58:55 -07:00
zephyr.py