zulip/zerver/lib
Mateusz Mandera f95dd628bd email: Fix race conditions with concurrent ScheduledEmail handling.
The main race conditions, which actually happened in production was with
concurrent execution of deliver_email and clear_scheduled_emails.
clear_scheduled_emails could delete all email.users in the middle of
deliver_email execution, causing it to pass empty to_user_ids list to
send_email. We mitigate this by getting the list of user ids in a single
query and moving forward with that snapshot, not having to worry about
database data being mutated anymore.

clear_scheduled_emails had potential race conditions with concurrent
execution of itself due to not locking the appropriate rows upon
selecting them for the purpose of potentially deleting them. FOR UPDATE
locks need to be acquired to prevent simultaneous mutation.

Tested manually with some print+sleep debugging to make some races
happen.

fixes #zulip-2k (sentry)
2020-09-09 15:59:01 -07:00
..
markdown python: Pre-fix a few spots for better Black formatting. 2020-09-03 17:51:09 -07:00
url_preview url_preview: Only extract img tags with an src. 2020-08-18 14:26:21 -04:00
webhooks python: Replace unnecessary intermediate lists with generators. 2020-09-02 11:15:41 -07:00
__init__.py
actions.py python: Tweak some magic trailing commas to avoid Black bugs. 2020-09-03 17:51:09 -07:00
addressee.py addressee: Remove redundant casts. 2020-06-22 17:11:42 -07:00
alert_words.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
attachments.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
avatar_hash.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
avatar.py python: Convert more percent formatting to Python 3.6 f-strings. 2020-06-14 23:27:22 -07:00
bot_config.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
bot_lib.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
bot_storage.py test_service_bot_system: Fix type: ignore issues. 2020-06-30 18:58:23 -07:00
bulk_create.py python: Replace unnecessary intermediate lists with generators. 2020-09-02 11:15:41 -07:00
cache_helpers.py sessions: Fix type: ignore issues. 2020-06-30 18:58:23 -07:00
cache.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
camo.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
ccache.py python: Fix misuse of Optional types for optional parameters. 2020-06-13 15:31:27 -07:00
context_managers.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
create_user.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
db.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
debug.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
dev_ldap_directory.py python: Convert more percent formatting to Python 3.6 f-strings. 2020-06-14 23:27:22 -07:00
digest.py python: Prefer kwargs form of dict.update. 2020-09-03 17:51:09 -07:00
display_recipient.py api: Remove (sender_)short_name from message APIs. 2020-07-17 11:15:14 -07:00
domains.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
email_mirror_helpers.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
email_mirror.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
email_notifications.py python: Prefer kwargs form of dict.update. 2020-09-03 17:51:09 -07:00
email_validation.py isort: Enable black profile. 2020-09-02 11:00:07 -07:00
emoji.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
error_notify.py Error reporting emails: Indicate user role. 2020-08-03 13:35:51 -07:00
event_schema.py python: Replace list literal concatenation with * unpacking. 2020-09-02 11:15:41 -07:00
events.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
exceptions.py exceptions: Raise InvalidSubdomainError when realm is invalid. 2020-09-03 10:45:14 -07:00
export.py python: Prefer --flag=option over --flag option. 2020-09-03 17:51:09 -07:00
external_accounts.py request: Improve validator type so mypy can check it against REQ. 2020-06-20 22:29:15 -07:00
fix_unreads.py pointer: Remove fix_pre_pointer fixing logic. 2020-06-18 12:55:59 -07:00
generate_test_data.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
home.py refactor: Extract build_page_params_for_home_page_load. 2020-07-22 17:15:03 -07:00
hotspots.py python: Pre-fix a few spots for better Black formatting. 2020-09-03 17:51:09 -07:00
html_diff.py html_diff: Really return str, not bytes. 2020-08-07 11:12:32 -07:00
html_to_text.py python: Replace None defaults with empty collections where appropriate. 2020-06-13 15:31:27 -07:00
i18n.py cache: Used lru_cache from functools instead of django.utils.lru_cache. 2020-08-14 11:34:04 -07:00
import_realm.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
initial_password.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
integrations.py integrations: Re-add Canarytoken integration. 2020-07-20 15:34:40 -07:00
logging_util.py settings: Adjust documentation and comment references to settings.py. 2020-08-24 13:13:16 -07:00
management.py python: Elide type=str from argparse arguments. 2020-09-03 16:17:14 -07:00
mdiff.py docs: Capitalize Markdown consistently. 2020-08-11 10:23:06 -07:00
mention.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
message.py python: Prefer kwargs form of dict.update. 2020-09-03 17:51:09 -07:00
migrate.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
mobile_auth_otp.py python: Replace unnecessary intermediate lists with generators. 2020-09-02 11:15:41 -07:00
name_restrictions.py python: Pre-fix a few spots for better Black formatting. 2020-09-03 17:51:09 -07:00
narrow.py message_fetch: Allow access to web-public msgs for unauth users. 2020-08-12 16:39:02 -07:00
onboarding.py onboarding: Move send_welcome_bot_response to onboarding. 2020-09-03 17:41:08 -07:00
outgoing_webhook.py python: Catch JSONDecodeError instead of ValueError when decoding JSON. 2020-08-12 11:59:59 -07:00
parallel.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
presence.py python: Tweak some magic trailing commas to avoid Black bugs. 2020-09-03 17:51:09 -07:00
profile.py decorator: Strengthen types of signature-preserving decorators. 2020-06-30 18:58:23 -07:00
push_notifications.py python: Prefer kwargs form of dict.update. 2020-09-03 17:51:09 -07:00
pysa.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
queue.py queue: Remove a no-longer-correct comment. 2020-08-14 11:30:13 -07:00
rate_limiter.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
realm_description.py refactor: Rename convert to markdown_convert. 2020-07-06 12:39:59 -07:00
realm_icon.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
realm_logo.py realm_logo: Return default logo for realms on LIMITED plan. 2020-06-18 17:46:26 -07:00
redis_utils.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
remote_server.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
request.py python: Catch JSONDecodeError instead of ValueError when decoding JSON. 2020-08-12 11:59:59 -07:00
response.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
rest.py rest: Simplify authentication error handling. 2020-08-30 14:51:50 -07:00
retention.py realm: Fix strict_optional errors. 2020-07-06 11:25:48 -07:00
send_email.py email: Fix race conditions with concurrent ScheduledEmail handling. 2020-09-09 15:59:01 -07:00
server_initialization.py database: Remove short_name from UserProfile. 2020-07-17 11:15:15 -07:00
sessions.py sessions: Fix type: ignore issues. 2020-06-30 18:58:23 -07:00
soft_deactivation.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
sqlalchemy_utils.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
statistics.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
storage.py docs: Merge front-end-build-process with html-css. 2019-10-23 13:08:36 -07:00
stream_recipient.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
stream_subscription.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
stream_topic.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
streams.py topic_history: Allow anonymous access in web-public streams. 2020-09-04 10:17:56 -07:00
subdomains.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
test_classes.py django_api, test_classes: Use Python 3 form of super(). 2020-09-02 11:31:46 -07:00
test_console_output.py tests: Add extra console output detection in test-backend output. 2020-08-27 11:39:53 -07:00
test_data.source.txt docs: Capitalize Markdown consistently. 2020-08-11 10:23:06 -07:00
test_fixtures.py python: Pre-fix a few spots for better Black formatting. 2020-09-03 17:51:09 -07:00
test_helpers.py tests: Add a mock to verify that queue_json_publish is valid JSON. 2020-09-03 17:34:31 -07:00
test_runner.py python: Pre-fix a few spots for better Black formatting. 2020-09-03 17:51:09 -07:00
tex.py python: Convert subprocess.Popen.communicate to run or check_output. 2020-09-03 17:42:35 -07:00
thumbnail.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
timeout.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
timestamp.py timestamp: Remove datetime_to_precise_timestamp for datetime.timestamp. 2020-07-31 22:10:14 -07:00
timezone.py docs: Capitalize Markdown consistently. 2020-08-11 10:23:06 -07:00
topic_mutes.py python: Tweak some magic trailing commas to avoid Black bugs. 2020-09-03 17:51:09 -07:00
topic.py topic: Reorder topic history functions. 2020-08-25 17:03:48 -07:00
transfer.py upload: Avoid fetching bucket objects repeatedly. 2020-06-18 17:55:13 -07:00
type_debug.py decorator: Replace type: ignore with cast, avoid Any. 2020-06-30 18:58:23 -07:00
types.py api: Remove (sender_)short_name from message APIs. 2020-07-17 11:15:14 -07:00
unminify.py python: Manually convert more percent-formatting to f-strings. 2020-06-14 23:27:22 -07:00
upload.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
url_encoding.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
user_agent.py user_agent: Compile the regular expression. 2020-02-14 10:26:37 -08:00
user_groups.py python: Replace unnecessary intermediate lists with generators. 2020-09-02 11:15:41 -07:00
user_status.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
users.py python: Replace list literal concatenation with * unpacking. 2020-09-02 11:15:41 -07:00
utils.py python: Use standard secrets module to generate random tokens. 2020-09-09 15:52:57 -07:00
validator.py python: Replace ujson with orjson. 2020-08-11 10:55:12 -07:00
widget.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00
zcommand.py python: Rewrite dict() as {}. 2020-09-02 11:15:41 -07:00
zephyr.py python: Sort imports with isort. 2020-06-11 16:45:32 -07:00