Commit Graph

9011 Commits

Author SHA1 Message Date
Tim Abbott
a6ca95dfc4 slack import: Fix all messages being imported to one channel.
This was an ugly variable-escape-from-loop regression introduced in
e59ff6e6db.
2018-12-12 17:54:37 -08:00
Vishnu Ks
8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott
ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Tim Abbott
d6217eb862 slack import: Fix empty values for custom profile fields.
The Slack import process would incorrectly issue
CustomProfileFieldValue entries with a value of "" for users who
didn't have a given CustomProfileField (especially common for the
"skype" and "phone" fields).  This had no user-visible effect, but
certainly added some clutter in the database.
2018-12-12 12:58:27 -08:00
Tim Abbott
5d5b1afd2d rate_limiter: Improve handling of deadlocks.
We should rate-limit users when our rate limiter deadlocks trying to
increment its count; we also now log at warning level (so it doesn't
send spammy emails) and include details on the user and route was, so
that we can properly investigate whether the rate-limiting on the
route was in error.
2018-12-12 12:19:01 -08:00
Tim Abbott
8a11c94a2d fakeldap: Add thumbnailPhoto/jpegPhoto for testing avatar syncing. 2018-12-12 11:23:33 -08:00
Tim Abbott
e9900b2bdf gitter: Do something reasonable with invalid fullnames. 2018-12-12 10:07:52 -08:00
Tim Abbott
27f3919259 push_notifications: Also treat ConnectionError as a retry.
It's a bug that the HTTP20 library isn't catching these, but we just
have to work around it.
2018-12-12 09:40:51 -08:00
Tim Abbott
914ae88c62 push_notifications: Expand APNS main try/except block.
We should be catching exceptions trying to send the notice as well as
in the result section.
2018-12-12 09:40:51 -08:00
Roman Godov
9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott
15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Tim Abbott
b7127a395b server_settings: Use a reasonable default when User-Agent not specified.
Apparently, some Zulip clients don't send a User-Agent in their
requests to /server_settings, and this makes that not 500.
2018-12-11 11:30:11 -08:00
Tim Abbott
2a39852e91 auth: Add logging for which client does /server_settings requests.
This is generally helpful for debugging.
2018-12-11 11:30:11 -08:00
Tim Abbott
88368996fb decorator: Rename remote_server_request to skip_update_user_activity.
This makes it reasonable to call process_client from other contexts,
like /server_settings, where we don't have an authenticated
UserProfile object.
2018-12-11 11:30:11 -08:00
Tim Abbott
14bfa74069 registration: Enable copying profile settings in production.
Now that we've styled this feature properly, this makes it possible to
copy various user-preferences type profile data in production when
making a new account with the same email address as an existing
account.
2018-12-10 16:55:07 -08:00
Max Nussenbaum
536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Jack Zhang
be9b6a6dee compatibility: Add a compatibility check to api_get_server_settings.
This should make it convenient for the mobile app to present errors of
the form "Your Zulip app is not new enough for this Zulip server".
2018-12-10 15:06:09 -08:00
Jack Zhang
8711fe7bae compatibility: Respond with HTTP 400 when no User-Agent header.
Prior to this commit, a request to /compatibility without a User-Agent
header would result in the server 500ing.
2018-12-10 15:04:34 -08:00
Tim Abbott
a1d99adce7 auth: Handle SSO_APPEND_DOMAIN in remote_user SSO for mobile.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.

Fixes #11005.
2018-12-10 10:36:15 -08:00
neiljp (Neil Pilgrim)
63768858ff dependencies: Upgrade mypy to version 0.650.
Broaden the type of the AbstractEnum __reduce_ex__ parameter to object; this
matches the parameter type specified in the latest enum.pyi file in typeshed.

Fixes #10996.
2018-12-09 22:20:12 -08:00
Marco Burstein
ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht
a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
rht
a0dbcde063 export_files_from_s3: Move saving s3 object to local file to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht
0ddb242583 export_files_from_s3: get s3 object info in dict to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht
1cecf0f142 export_files_from_s3: Move checking for s3 oject's metadata to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
Joshua Pan
441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott
5c2e7a169f test_events: Use a better assertion when comparing event count. 2018-12-06 17:43:26 -08:00
Tim Abbott
7e6f30caab actions: Extract send_user_email_update_event.
This will make it easy for us to reuse this event-sending logic for
the email_address_visibility change code path.
2018-12-06 16:26:46 -08:00
Tim Abbott
e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott
9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Rishi Gupta
82ee496efc webhooks/stripe: Update fixture for customer_subscription_created. 2018-12-06 11:29:16 -08:00
Rishi Gupta
76aebac568 webhooks/stripe: Format timestamps as dates. 2018-12-06 11:29:16 -08:00
Rishi Gupta
027d5e90c5 webhooks/stripe: Update how we handle updated events.
Also more explicitly ignores the Stripe Connect related events in the
'account' category.
2018-12-06 11:29:16 -08:00
Rishi Gupta
5fb683e788 webhooks/stripe: Use exceptions instead of json_success to exit main body. 2018-12-06 11:29:16 -08:00
Tim Abbott
fc1c146d31 export: Remove assertion on current working directory.
This command hasn't made deep assumptions about CWD for a long time,
and this enables users to run it through a symlink (etc.).

Fixes #10961.
2018-12-06 11:05:40 -08:00
Tim Abbott
9da4b58c8e docs: Fix /help/ sidebar being accidentally missing.
Apparently, we didn't have a test for this, and it was broken in a
recent refactoring.
2018-12-06 10:13:51 -08:00
Vishnu Ks
18f90754da portico: Change wording of finding organization in /accounts/go. 2018-12-06 09:24:22 -08:00
Tim Abbott
460249704e send_email: Fix cleanup for incorrectly migrated jobs.
This logic, unfortunately, had a typo.
2018-12-05 16:30:28 -08:00
Tim Abbott
1adc40f014 import: Deduplicate functions for uploading to S3/files.
We've had a long stream of bugs existed because only one of these two
code paths was tested (usually the local uploads backend).  By
deduplicating these functions, we ensure that this category of bugs no
longer happens.

Following my recent refactor, this is just a straightforward merge,
with code for one or the other backend ending up inside an if
statement.
2018-12-05 16:15:01 -08:00
Tim Abbott
c9b801efde import: Use the s3_path attribute for path_maps unconditionally.
While the s3_path is almost always the same as the path, structurally,
`path` is the location in the export object, whereas s3_path is the
URL path.
2018-12-05 16:15:01 -08:00
Tim Abbott
f4c5a45f4f import: Fix S3 paths for imported avatar PNG.
Previously, we were incorrectly importing avatar PNGs to a filename
without the .png extension, resulting in them effectively not being
imported.

This was mitigated by the fact that we imported the originals and ran
the appropriate `ensure_` functions, but still a bug.
2018-12-05 16:15:01 -08:00
Tim Abbott
412dc8dcda import: Set last_modified in import_uploads_local.
This has no effect other than to make the S3 and local code paths more
nearly identical.
2018-12-05 16:15:01 -08:00
Tim Abbott
d8d0492d64 import: Restructure uploads path logic to be more similar.
This is preparation for future deduplication of the two redundant
uploads backends.
2018-12-05 16:15:01 -08:00
Tim Abbott
671ceccd78 import: Deduplicate medium avatars special logic.
This requires a bit of care with upload_backend to avoid breaking how
we mock that class in our tests.
2018-12-05 16:15:01 -08:00
Tim Abbott
36b43a6d7a import: Deduplicate first block of import_uploads logic. 2018-12-05 16:15:01 -08:00
Tim Abbott
f80bab58c0 import_realm: Add progress indicator for importing uploads.
This makes it easier to see how we're doing when uploading a very
large number of files.
2018-12-05 16:15:01 -08:00
Tim Abbott
9e06ab45bf webhooks: Fix HttpResponse with notify_bot_owner_on_invalid_json.
Apparently, there was a bug in notify_bot_owner_on_invalid_json, where
we didn't reraise the JsonableError.

We fix this with a refactoring that makes the exception layering
clearer as well.
2018-12-05 15:17:31 -08:00
Tim Abbott
9de1bd44e2 compatibility: Clean up linter logic for too-old client. 2018-12-05 15:04:21 -08:00
Greg Price
32c6b81b3a compatibility: Add test data from real server logs.
In a quick scan of today's nginx logs on chat.zulip.org, there
were 20 distinct user-agents that begin with 'ZulipMobile/'.
Here's a representative sampling of them, such that the rest
were all boringly similar to one of these.

First, to make room for these without an excess of copy-paste and
overlong lines, convert this test to a data-oriented style.  The
existing, synthetic cases appear in the new data followed by the
seen-in-the-wild cases.

Happily, the code being tested passes all these new cases unchanged.
2018-12-05 14:58:13 -08:00
Greg Price
fb7bfbe9ab compatibility: Require version 16.2.96+ for our Android app.
This release is from 2018-08-22, a little over 100 days ago.

It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null".  We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.

The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
2018-12-05 14:58:13 -08:00