CodeCow/zulip
1
0
Fork 0
mirror of https://github.com/zulip/zulip.git synced 2026-07-09 21:21:47 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
54c01ac46e
zulip/zerver/lib
History
Zixuan James Li a081428ad2 user_groups: Make locks required for updating user group memberships. 
**Background**

User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.

This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.

**Solution**

To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.

The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.

Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:

1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.

For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.

**Error handling**

We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.

**Notes**

An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.

User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.

**Testing**

We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).

To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.

The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).

**Security notes**

get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.

We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
2023-08-24 17:21:08 -07:00
..
markdown emoji: Match emoji sequences in markdown. 2023-08-23 16:18:15 -07:00
upload ruff: Fix E721 Do not compare types, use isinstance(). 2023-08-17 17:05:34 -07:00
url_preview ruff: Fix UP032 Use f-string instead of format call. 2023-07-19 16:14:59 -07:00
webhooks ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
__init__.py
addressee.py users: Directly access id of foreign keys instead of full object. 2023-07-20 10:44:39 -07:00
alert_words.py alert_words: Refactor the code to flush alert_words cache. 2023-06-28 18:03:32 -07:00
async_utils.py requirements: Upgrade Python requirements. 2022-05-03 10:10:06 -07:00
attachments.py upload: Rename delete_message_image to use word "attachment". 2023-03-02 16:36:19 -08:00
avatar_hash.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
avatar.py settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
bot_config.py ruff: Fix UP032 Use f-string instead of format call. 2023-08-02 15:58:55 -07:00
bot_lib.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
bot_storage.py python: Import F, Q, QuerySet from their canonical module. 2023-03-05 14:46:28 -08:00
bulk_create.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
cache_helpers.py models: Fetch "recipient" object when along with "Huddle" object. 2023-08-10 17:35:43 -07:00
cache.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
camo.py typing: Apply trivial none-checks with assertions as necessary. 2022-06-23 19:25:48 -07:00
ccache.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
compatibility.py ruff: Fix DTZ004 datetime.datetime.utcfromtimestamp(). 2023-01-04 16:25:07 -08:00
context_managers.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
create_user.py models: Add a unique index on UserProfile.api_key. 2023-05-19 11:11:04 -07:00
data_types.py ruff: Fix more of RUF010 Use conversion in f-string. 2023-06-06 14:58:11 -07:00
db.py python: Annotate type aliases with TypeAlias. 2023-08-07 10:02:49 -07:00
debug.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
default_streams.py streams: Pass stream_weekly_traffic field in stream objects. 2023-08-06 18:06:42 -07:00
dev_ldap_directory.py tests: Update tests to use example profile picture. 2022-10-31 14:36:54 -07:00
digest.py python: Annotate type aliases with TypeAlias. 2023-08-07 10:02:49 -07:00
display_recipient.py mypy: Improve type checks for user display recipients. 2023-08-10 18:13:43 -07:00
domains.py python: Convert deprecated Django ugettext alias to gettext. 2021-04-15 18:01:34 -07:00
drafts.py drafts: Access recipient_id when creating or editing Draft objects. 2023-08-10 17:35:43 -07:00
email_mirror_helpers.py email_mirror: Move ZulipEmailForwardUserError into email_mirror_helpers. 2021-08-31 16:37:54 -07:00
email_mirror.py mypy: Improve type checks for user display recipients. 2023-08-10 18:13:43 -07:00
email_notifications.py email-templates: Add zulip_onboarding_topics email templates. 2023-08-18 16:25:48 -07:00
email_validation.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
emoji_utils.py emoji: Match emoji sequences in markdown. 2023-08-23 16:18:15 -07:00
emoji.py python: Convert translated positional {} fields to {named} fields. 2023-07-18 15:19:07 -07:00
event_schema.py events: Add test to remove existing value for custom profile field. 2023-08-07 11:39:27 -07:00
events.py typing: Add typing constants to the post register api response. 2023-08-23 16:36:44 -07:00
exceptions.py reactions: Add error code for duplicate addition/removal. 2023-07-19 16:18:31 -07:00
export.py migration: Rename extra_data_json to extra_data in audit log models. 2023-08-16 17:18:14 -07:00
external_accounts.py ruff: Fix SIM118 Use k not in d instead of k not in d.keys(). 2023-07-24 10:39:28 -07:00
fix_unreads.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
generate_test_data.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
github.py ruff: Fix RSE102 Unnecessary parentheses on raised exception. 2023-02-04 16:34:55 -08:00
home.py narrow: Split out narrow_helpers. 2023-06-30 11:26:23 -07:00
hotspots.py ruff: Fix SIM118 Use key in dict instead of key in dict.keys(). 2023-01-04 16:25:07 -08:00
html_diff.py html_diff: Fix lxml import. 2023-03-05 14:46:28 -08:00
html_to_text.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
i18n.py django: Use HttpRequest.headers. 2022-05-13 20:42:20 -07:00
import_realm.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
initial_password.py initial_password: Add explicit development environment assertion. 2022-03-21 12:05:59 -07:00
integrations.py python: Annotate type aliases with TypeAlias. 2023-08-07 10:02:49 -07:00
logging_util.py ruff: Fix PLR1714 Consider merging multiple comparisons. 2023-07-23 15:21:33 -07:00
management.py send_custom_email: Stop turning every user query into an id-based set. 2023-08-09 15:49:49 -07:00
mdiff.py node_tests: Move to web/tests. 2023-02-23 16:04:17 -08:00
mention.py mention: Determine @topic mention during message rendering. 2023-07-13 11:34:48 -07:00
message.py message: Do not pass "sender__realm" to select_related. 2023-08-23 11:38:32 -07:00
migrate.py typing: Replace CursorObj by CursorWrapper. 2021-08-20 05:54:19 -07:00
mobile_auth_otp.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
muted_users.py muted users: Make file naming consistent. 2023-02-10 15:39:57 -08:00
name_restrictions.py name_restrictions: Update disposable_email_domains usage. 2023-07-19 16:14:59 -07:00
narrow_helpers.py narrow: Split out narrow_helpers. 2023-06-30 11:26:23 -07:00
narrow.py Revert "narrow: Fix topic highlighting issue with apostrophes in search results." 2023-08-15 17:51:03 -07:00
notes.py notes: Separate __notes_map per-subclass. 2022-10-10 08:42:13 -07:00
notification_data.py notifications: Rename 'pm' to 'dm' in 'RecipientInfoResult' dataclass. 2023-08-10 17:41:49 -07:00
onboarding.py message: Access realm from SendMessageRequest object directly. 2023-08-23 11:38:32 -07:00
outgoing_http.py python: Replace requests.packages.urllib3 alias with urllib3. 2022-01-23 22:14:17 -08:00
outgoing_webhook.py outgoing_webhook: Respect settings.OUTGOING_WEBHOOK_TIMEOUT_SECONDS. 2023-05-16 07:00:37 -07:00
per_request_cache.py per-request caches: Add per_request_cache library. 2023-08-11 11:09:34 -07:00
presence.py presence: Support null values in UserPresence. 2023-04-26 14:26:47 -07:00
profile.py profile: Strengthen decorator types using ParamSpec. 2022-04-14 12:44:35 -07:00
push_notifications.py push notifications: Use hex_codepoint_to_emoji. 2023-08-23 16:18:15 -07:00
pysa.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
queue.py bots: Remove private stream subscriptions on changing bot owner. 2023-08-16 15:37:37 -07:00
rate_limiter.py ruff: Fix PERF102 Using only the keys/values of a dict. 2023-08-07 17:23:55 -07:00
realm_description.py markdown: Refactor out additional properties added to Message. 2021-06-24 18:14:53 -07:00
realm_icon.py settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
realm_logo.py settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
recipient_users.py models: Remove get_huddle_recipient and use get_or_create_huddle. 2023-08-10 17:35:43 -07:00
redis_utils.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
remote_server.py python: Convert translated positional {} fields to {named} fields. 2023-07-18 15:19:07 -07:00
request.py ruff: Fix FLY002 Consider f"…" instead of string join. 2023-08-07 17:12:41 -07:00
response.py webhooks: Use 200 status code for unknown events. 2023-07-11 13:51:37 -07:00
rest.py ruff: Fix RSE102 Unnecessary parentheses on raised exception. 2023-02-04 16:34:55 -08:00
retention.py retention: Do not archive attachments with scheduled messages. 2023-08-06 13:40:02 -07:00
safe_session_cached_db.py session: Enforce that changes cannot happen in a transaction. 2022-03-15 13:52:15 -07:00
scheduled_messages.py scheduled_messages: Add reasonable failure handling. 2023-05-09 13:48:28 -07:00
scim_filter.py scim: Order Users by id when queried using filter syntax. 2021-11-26 16:06:16 -08:00
scim.py users: Set tos_version to -1 for users who have not logged-in yet. 2023-05-16 13:52:56 -07:00
send_email.py send_email: Use a consistent order when sending custom emails to users. 2023-08-23 10:49:34 -07:00
server_initialization.py auth: Rewrite data model for tracking enabled auth backends. 2023-04-18 09:22:56 -07:00
sessions.py typing: Add none-checks for miscellaneous cases. 2022-05-31 09:43:55 -07:00
singleton_bmemcached.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
soft_deactivation.py notification_trigger: Rename private_message to direct_message. 2023-08-10 17:41:49 -07:00
sounds.py actions: Split out zerver.lib.sounds. 2022-04-14 14:26:40 -07:00
sqlalchemy_utils.py sqlalchemy_utils: Remove NonClosingPool.recreate override. 2022-02-10 11:59:41 -08:00
storage.py ruff: Fix PLE0101 Explicit return in __init__. 2023-02-23 11:47:08 -08:00
stream_color.py streams: Extract stream_color library. 2022-03-14 18:01:36 -07:00
stream_subscription.py topic_mentions: Fetch users to be notified of @topic mentions. 2023-07-13 11:34:48 -07:00
stream_topic.py message_send: Handle notifications for UNMUTED topic in a muted stream. 2023-03-06 19:15:45 -08:00
stream_traffic.py stream_traffic: Update get_streams_traffic to return None for zephyr realm. 2023-08-21 15:21:58 -07:00
streams.py stream_traffic: Update get_streams_traffic to return None for zephyr realm. 2023-08-21 15:21:58 -07:00
string_validation.py python: Convert translated positional {} fields to {named} fields. 2023-07-18 15:19:07 -07:00
subdomains.py settings: Allow customization of STATIC_URL. 2023-02-14 17:17:06 -05:00
subscription_info.py streams: Don't compute traffic data for sub objects in zephyr realm. 2023-08-21 15:21:58 -07:00
templates.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
test_classes.py user_groups: Make locks required for updating user group memberships. 2023-08-24 17:21:08 -07:00
test_console_output.py requirements: Upgrade Python requirements. 2023-04-25 21:20:33 -07:00
test_data.source.txt Rename default branch to ‘main’. 2021-09-06 12:56:35 -07:00
test_fixtures.py ruff: Fix ANN204 missing return type annotation for __init__. 2022-11-16 09:29:11 -08:00
test_helpers.py user_groups: Make locks required for updating user group memberships. 2023-08-24 17:21:08 -07:00
test_runner.py python: Annotate type aliases with TypeAlias. 2023-08-07 10:02:49 -07:00
tex.py python: Replace universal_newlines with text. 2022-01-23 22:16:01 -08:00
thumbnail.py docs: Remove some outdated references to thumbnailing.md doc. 2022-07-12 17:44:24 -07:00
timeout.py test_timeout: Skip test_timeout_warn on Python 3.11 for coverage issue. 2023-05-18 11:52:22 -07:00
timestamp.py timestamp: Switch to a slightly faster datetime_to_timestamp. 2023-02-23 12:15:13 -08:00
timezone.py timezone: Improve tzdata parser’s compatibility with zic(8). 2022-09-20 16:58:31 -07:00
topic.py topic: Set a max batch_size on bulk_upate call. 2023-08-14 13:33:20 -07:00
transfer.py uploads: Allow uploads to set storage class. 2023-07-19 16:19:34 -07:00
types.py python: Annotate type aliases with TypeAlias. 2023-08-07 10:02:49 -07:00
url_encoding.py narrow urls: Avoid complicated optional types. 2023-08-10 18:13:43 -07:00
url_redirects.py help: Rename edit-or-delete-a-message.md and update links. 2023-08-22 14:50:23 -07:00
user_agent.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
user_counts.py actions: Split out zerver.lib.user_counts. 2022-04-14 17:14:30 -07:00
user_groups.py user_groups: Make locks required for updating user group memberships. 2023-08-24 17:21:08 -07:00
user_message.py actions: Split out zerver.lib.user_message. 2022-04-14 17:14:30 -07:00
user_status.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
user_topics.py user_topics: Update 'topic_has_visibility_policy' to support INHERIT. 2023-04-18 16:40:57 -07:00
users.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
utils.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
validator.py python: Convert translated positional {} fields to {named} fields. 2023-07-18 15:19:07 -07:00
widget.py ruff: Fix B034 re.split, re.sub should pass keyword arguments. 2023-07-19 16:14:59 -07:00
zcommand.py ruff: Fix UP032 Use f-string instead of format call. 2023-08-02 15:58:55 -07:00
zephyr.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00