As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.
cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
Outgoing email documentation should mention that the SMTP server needs
to allow emails originating from ZULIP_ADMINISTRATOR as well as the
noreply email addresses; previously, one would typically find this out
using the test tool, which was not ideal.
Substantially tweaked by tabbott for organization and content.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend. These are:
* GoogleMobileOauth2Backend, the ancient code path for mobile
authentication last used by the EOL original Zulip Android app.
* The `finish_google_oauth2` code path in zerver/views/auth.py, which
was the webapp (and modern mobile app) Google authentication code
path.
This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
We had an organization with engineers for whom English was not their
native language think they needed to go through this laborious
process, which was definitely counterproductive for them.
Many Linux distros like Debian Buster, Fedora 29, etc. have a policy
where any services installed are disabled by default. So we should let
the developer know that they should explicitly enable and start the
Docker daemon.
Also, we should be verbose on how to do vagrant up using Docker as a
provider specifically (and similar for Virtualbox).
This was rewritten by tabbott to use markdown steps, avoid
distractions about things you don't need to change, and remove the
screenshots, which I felt were more clutter than value.
We had several patches to spectrum, but the only essential one
(0ea770fc18) had already been fixed upstream,
and another was just handling jQuery deprecation warnings for not yet removed features.
See #12749 for details.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It's not actually a reasonable experience to use Zulip without working
full-text search, so we should more strongly direct folks to not do
it.
Fixes#12724.
Moving bootstrap-typeahead from bundles/commons.js to bundles/app.js
and csrf.js from bundles/app.js to bundles/commons.js makes
bundles/commons.js equivalent to the "common" bundle, so we can
replace the latter with the former.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser. Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.
This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.
This reverts commit b546391f0b (#1148).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This was only used in Ubuntu 14.04 Trusty.
Removing this also finally lets us simplify our security model
discussion of uploaded files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We only use it in this one place, so a comment right here seems the
most discoverable place to put it. If we started using it more...
probably the section in docs/documentation/overview.md about the
dev/sysadmin docs system should split off into a new file, and this
info would become a subsection there.
This provides a better entrypoint for developers to learn about
internationalization in Zulip without cluttering the article for
translators.
I also took the opportunity to add a proper for-developers
introduction, including a link to the very nice EdX guide on the
topic.
The `AUTH_LDAP_ALWAYS_UPDATE_USER` is `True` by default, and this would sync the
attributes defined in the `AUTH_LDAP_USER_ATTR_MAP` to the user profile. But,
the default code in `django-auth-ldap` would work correctly only for `full_name`
field. This commit disables the setting by default, in favour of using the
`sync_ldap_user_data` script as a cron job.
Move docs/tutorials/documenting-api-endpoint.md to
docs/documentation/api.md.
This makes it easier to find when browsing the complete set of
materials on writing Zulip documentation.
This also means the default dev environment is now based on Ubuntu
18.04 (bionic), part of our overall effort to migrate off Ubuntu
Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This guide was originally written as part of the process of adding
i18n support to Zulip, so it had way too much focus on development
tooling details not relevant to translators.
There's probably a separate follow-up project we should do to move the
developers-only content to a separate article, since it's kinda lost
here for that audience.
Thanks to David Wood for asking a question that inspired me to do
this.
We now use a Proxy to wrap zjquery elements, so
that we can detect callers trying to invoke methods
(or access attributes) that do not exist. We try
to give useful error messages in those cases.
The main impact here is that we force lots of tests
to explicitly stub `length`.
Also, we can't do equality checks on zjquery
objects any more due to the proxy object, but the
easy workaround is to compare selectors. (This
is generally an unnecessary technique, anyway.)
The proxy wrapper is fairly straightforward, and
we just have a few special cases for things like
"inspect" that happen when you try to print out
objects.
The numbers are in parens to avoid the markdown processor interpreting them
as numbers -- I couldn't get the automatic numbering to work out with the
paragraph breaks and so on, and this is probably good enough for now.
This commit renames the `create_stream_permission` field in the
templates to `create_stream_policy`, matching the field used in the
database model. This matches what `invite_to_stream_policy` does and
will be clearer when the `waiting_period_threshold` is split into its
own field.
We need to disable "CREATE EXTENSION pgroonga" in zulip-puppet-apply
by creating /usr/share/postgresql/10/pgroonga_setup.sql.applied.
Because PostgreSQL 10 isn't running in this case. If PostgreSQL 10
isn't running, we can't run "CREATE EXTENSION pgroonga".
We can't use pg_upgrade with PGroonga. PGroonga's install SQL
https://github.com/pgroonga/pgroonga/blob/master/data/pgroonga.sql has
conditions to support multiple PostgreSQL versions. So it's not safe
to use pg_upgrade. pg_upgrade copies metadata for PostgreSQL 9.5 to
PostgreSQL 10. We need to use pg_dump and pg_restore to upgrade
PGroonga correctly for PostgreSQL 10.
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.5-266.12: error: Element "commit" not allowed as child of element "em" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.13-266.17: error: End tag "em" violates nesting rules.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.69-211.136: error: Start tag "a" seen but an element of the same type was already open.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.171-211.174: error: Stray end tag "a".
"file:/srv/zulip/docs/_build/html/translating/hindi.html":252.36-252.48: info warning: Text run is not in Unicode Normalization Form C.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Element "method" not allowed as child of element "p" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":471.122-471.125: error: End tag "p" seen, but there were open elements.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Unclosed element "method".
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It's not required on Ubuntu Xenial (having been replaced by systemd)
and causes problems when installing/upgrading other packages; this
change matches a similar block of code in our installer.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
One longstanding gap in our production documentation is how to
properly do an upgrade to the operating system on which Zulip is
installed.
This adds that documentation.
Ideally, we'd get a few folks to test this procedure over the next few
days to make sure it's bulletproof.
Fixes#1705.
Fixes#10796.
There are two 'etc' words in the same line. This removes one of them.
Note from Tim: The old text was arguably correct, but removing it is definitely clearer.
Add references to TypeScript in documentation where appropriate, such
as in example bash commands and discussions of the file structure.
Add a new section to the Reading List with TypeScript resources.
Also update `.editorconfig` to support ".ts" files.
Fix part of #12000.
Two variables were declared and assigned the respective values of the
default settings for the system. If the keyword is used in the
~/.zulip-vagrant-config file, the value is assigned to the variable.
There is no straightforward way to customize the virtual machine's
number of cpus or memory, this commit addresses that fact.
This adds numerous manual testing methods and checks to the
`manual-testing.md` to verify the functioning of various hotkeys,
click handlers and appearance of the message box.
Sphinx/ReadTheDocs supports automatically translating links written as
to `.md` files to point to the corresponding `.html` files, so this
migration does not change the resulting HTML output in ReadTheDocs.
But it does fix apparent broken links on GitHub.
This doesn't prevent people from reading the documentation on GitHub
(so doesn't mitigate the fact that some rtd-specific syntax does not
render properly on GH), but it will prevent us from getting erroneous
issues reported about the hyperlinks not working.
Fixes: #11087.
The docs specify passing hostname with the --hostname flag, which
doesn't match the actual usage in scripts/setup/setup-certbot. This
change fixes the docs to match the actual usage.
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.
Fixes#11824.
Add the recommendation in docs for not using GitHub Desktop client for
zulip code as it has a bug of keeping the symlinks disabled and ignoring
any configuration done to re-enable it resulting into failure to identify
the received symlinks leading to failing test cases in the zulip development
environment.
This documentation had a tendency to bitrot, and in any case now that
we have tooling for doing Fedora (etc.) from provision, it's likely
the case that adding other Linux/UNIX distros we care about to
provision would not be difficult and is a better path than maintaining
this manually-curated duplicate of `tools/provision`.
Further, even if this documentation was maintained, one would still
end up wanting to run `provision` after rebasing a branch, so it was
never particularly practical for extended development.
The history of this was that there was a period where half of Vagrant
releases were broken (for everyone, e.g. downloading a base container
didn't work). It seems Vagrant has cleaned up their act at this point.
We really just want 2.0.x, not a specific version, and the direct
links we had are now quite old.
Thanks to Jonathon Hinchley for reporting this.
Fixes#11836.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.
No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
help.github.com seems to have a bug where HEAD on a redirected page
returns 404. This causes tools/test-documentation to fail. Fix it by
skipping the redirects.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This optimizes test-backend by skipping webhook
tests when run in default mode.
Tweaked by tabbott to extend the documentation and update the CI
commands.
Eventually, we'll just make this the default, but for now, we should
fix the typos and use our standing approach for documenting running
management commands.
This adds reference for reloading nginx when the certificates are
replaced so that the server works with the new certificates instead of
the old ones.
Fixes: #4849.
This endpoint serves requests which might originate from an image
preview link which had an http url and the message holding the image
link was rendered before we introduced thumbnailing. In that case
we would have used a camo proxy to proxy http content over https and
avoid mix content warnings.
In near future, we plan to drop use of camo and just rely on thumbor
to serve such images. This endpoint helps maintain backward
compatibility for links which were already rendered.
This setting splits away part of responsibility from THUMBOR_URL.
Now on, this setting will be responsible for controlling whether
we thumbnail images or not by asking bugdown to render image links
to hit our /thumbnail endpoint. This is irrespective of what
THUMBOR_URL is set to though ideally THUMBOR_URL should be set
to point to a running thumbor instance.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.
To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
Apparently, when we renamed these files to no longer have a .txt
extension, we accidentally removed them from the set of strings for
translation, because `manage.py makemessages` by default only
processes .txt and .html files under the templates/ directory.
Fix this by adding a .txt extension.
Bootstrap's typeahead is the main part of the project that we've
forked, and moving it to its own module should help unlock our ability
to upgrade bootstrap itself.
To support this, we add a pass_targets option to the main linter
library, because with current mypy, it's generally counterproductive
to pass the list of files in (can produce spurious errors; isn't
faster).
The testing section is more appropriate, since it's fundamentally part
of our CI system.
While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.
This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.
Fixes#286.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.
Fixes#7423.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.
Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
notifications settings are requested.
Fixes#8051.
A bunch of details were out of date about the current state of Zulip
and thus what projects make sense; while this isn't likely to be our
final 2019 ideas list, this should help avoid confusion about what's
happening.
A key part of this is the new helper, get_user_by_delivery_email. Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.
Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
This is required in some AWS regions.
The right long-term fix is to move to boto3 which doesn't have this
problem.
Allows us to downgrade the priority of #9376.
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:
https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/Fixes#10674.
Older versions of virtual box were giving installation error in new
MacOS Mojave. While originally we considered changing the docs to
point to the current version 5.2.20, it seems better to just not pin a
version.
Significantly tweaked by tabbott.
This is a preparator refactor for supporting hosting different Tornado
processes on different servers; to look up which Tornado server we
should be sending the event to, we'll need the realm object.
On OSX, the user id and group id don't match. So while the previous
code was always wrong, it produced incorrect output there. We can fix
this by replacing `whoami` with `id -g` for finding the current user's
group ID.
Tweaked by tabbott to move most of the content into the remote
development documentation, both for brevity in the main docs as well
as clarity.
Fixes#10694.
We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
Surprisingly hard to find a good base example of a user doc. This one should
at least give the basic pointer to sidebar_index.md and where the new file
goes.
In particular, this improves:
* The explanation of how data is mapped into Zulip
* The explanation of what is printed out by `manage.py query_ldap`
* Makes sure users create their first account with EmailAuthBackend.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond. The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.
Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface. Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username. And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.
It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
There are several situations in which we want to create a Customer and
stripe.Customer object before we really have a billing relationship with a
customer. The main one is giving non-profit or educational discounts.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.
This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).
The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.
Tweaked by tabbott to remove some additional JS code and update the
changelog.
Now that we have nice documentation for our export/import tools, we've
been seeing a lot of users trying to use that as their primary backup
process. Let's correct this.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.
This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.
We include a partial index on UserMessage for this flag.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.
Fixes part of #10042.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.
This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.
For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.
History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org. A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.
Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.
Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
It appears Luke's Dropbox folder/shared link that used to host some
tsearch_extras binaries was removed. It wasn't very high-value
regardless, because most of the platforms involved are deprecated, the
ones that don't generally have a PPA, and building from source is
pretty easy. So, we just remove these options from the documentation.
While we're at it, make clear we only support direct installation on
Ubuntu LTS.
Fixes#9863.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
As detailed in the documentation changes, this simplifies the
development workflow for doing UI work on the /stats pages.
The cost is a ~10% increase the time it takes to run `populate_db`,
which doesn't happen very often (and for most purposes manifests as a
1% increase in the time it takes to rebuild the database from scratch).
This saves about 400ms when running clean-unused-caches, basically by
calling its sub-rountines by import (rather than
`subprocess.check_call()`). The performance optimization seems well worth it.
Fixes#9766.
I don't think this is exactly the right place to document this, but
I'm not sure there's a better one without some restructuring this page
in general (which would probably have value).
Fixes#8769.
I only renamed references that I thought were absolutely necessary
and only if the resulting sentence structure wasn't awkward.
If the renaming resulted in awkward structure, I replaced the term
"webhook" with "integration" (but only in some very obvious cases).
Fixes#9500.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.
We also add a bucket policy, which is part of what one would want to
use this in production.
This addresses much, but not all, of #9361.
We don't reference this anymore (it was only ever used by the Dropbox
integration, which was hardcoded-off for years before being removed in
e6833b6427)
This fixes exceptions when sending PMs in development (where we were
trying to connect to the localhost push bouncer, which we weren't
authorized for, but even if we were, it wouldn't work, since there's
no APNS/GCM certs).
At the same time, we also set and order of operations that ensures one
has the opportunity to adjust the server URL before submitting
anything to us.
This will likely change pretty quickly as we build this feature; I
wrote this just to have a central place to keep track of what we'll
need to document when we ship this feature.
We started doing this for install docs in de2a2d0df, because `latest`
wasn't suitable and because I didn't know about readthedocs's `stable`
feature. The result has been that even with a checklist item, we
don't reliably update the link.
Instead, use the special `stable` version identifier on readthedocs to
link automatically to the highest version it knows about.
This should make it easier to find the templates that are actually
part of the core webapp, instead of having them all mixed together
with the portico pages.
In the very early days of Zulip, we didn't have unread counts; just
the pointer, and the correct behavior when opening a new tab was to
place you near the pointer. That doesn't make any sense now that we
do have unread counts, and this corner case has been a wart for a long
time.
This commit does the main behavior change here. However, there's a
bug we need to fix, where we might end up trying to pre-render a view
of the narrow based on the `all_msg_list` data before `all_msg_list`
is caught up). We need to fix that bug before we can merge this; it
should be possible to determine that using `FetchStatus` on
`all_msg_list`, or with better performance by using the `unread_msgs`
structure to determine whether the message we should be selecting is
present locally.
Fixes#789.
Fixes#9070.
This is a first pass at fixing node docs. This commit eliminates
some text that is either obsolete or just overly confusing, and
it fixes some of the code samples to reflect how the API has
evolved in the last couple years. We also prominently tell
you how to run the tests.
I don't think anybody ever really used this feature, which I
developed but don't even use myself. It kind of runs counter
to the minimalist approach of the rest of node tests.
I would eventually like to re-think the template tests altogether.
They're slow, and we could solve that somewhat by replacing
jsdon/jquery with an HTML parser library to verify structural
things.
It's also possible that we can just rely on our template linters
to catch the biggest class of errors (malformed tags) and let
code review do the rest.
And it's also possible that we should make a second attempt to
ramp up tooling on making it easy to verify templates, but it
doesn't have to be part of the node tests. If we did that, we
would also potentially use tooling for Python-side templates.
These aren't perfect -- in particular "core chat experience" can
probably be broken up -- but I think they help in making a quick skim
work for getting some sense of what the changes are.
This change just reorders and adds headings, with virtually no wording
changes.
This is kind of easy to gloss over, especially with the framing
as a "format"; surely if things work at all, the file format
must have been right, right? It's really a bit more substantive
than that; say so and also add a bit more description.
In addition to many small edits for formatting and clarity, a few more
significant changes:
* In the main instructions, refer specifically to restarting the
server and to testing that the config works.
* Add SendGrid to the recommended list, as it seems like people
give it a somewhat stronger reputation these days than Mailgun.
* Discuss EMAIL_USE_TLS and EMAIL_PORT along with host, user, and
password in the "free services" section. Though those bullets feel
kind of duplicative to me already.
Let's get right to the point of how to configure SMTP once you know
what you want. That section is pretty short anyway; and we can have
a first step direct the reader to our suggestions if they don't know
what service they want to use.
Also adjust the hierarchy of the headings: group the various
alternative email services under one heading, and group
troubleshooting together under an independent heading.
Also correct what we say about EMAIL_PORT: the Django default is
apparently 25, so if the provider *does* use the usual port 587
then we'll need the port to be set.
In our new system for updating realm settings, we don't need to create
separate functions to update the input elements for each feature.
Most of the work is done automatically by
`settings_org.sync_realm_settings`.
Usually, to debug a small change, you have to remove some tests from JSON
because of lack of support for comments in JSON. This commit allows to
ignore some tests by setting `"ignore" : true` in the bugdown fixtures.
Also, since this is only for while developing, the complete test suite will
throw an error if we leave an 'ignored' test in a commit.
This will allow realm admins to remove others from private stream to
which the realm administrator is not subscribed; this is important for
managing those streams, because previously nobody could remove users
from private streams that didn't have any realm administrators
subscribed.
This will allow realm admins to access subscribers of unsubscribed
private stream. This is a preparatory commit for letting realm admins
remove those users.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
The main point of this change is to streamline the core
code for input pills, and we use also modify user groups.
The main change to input_pill.js is that you now
configure a function called `create_item_from_text`, and
that can return an arbitrary object, and it just needs
a field called `display_value`.
Other changes:
* You now call `input.create(opts)` to create the
widget.
* There is no longer a cache, because we can
write smarter code in typeahead `source` functions
that exclude ids up front.
* There is no value/optinalKey complexity, because
the calling code can supply arbitrary objects and
do their own external data management on the pill
items.
* We eliminate `prependPill`.
* We eliminate `data`, `keys`, and `values`, and just
have `items`.
The summary already has a qualifier that basically says it shouldn't
matter for most people -- making it simultaneously the most
complicated bullet there, and among the least likely to matter.
And in fact, this requirement shouldn't matter for *anyone* when first
experimenting with Zulip. If certbot won't work in a given admin's
environment, and the available ways to get a cert aren't convenient,
they can always let the installer generate a self-signed cert to get
going, and circle back to the issue later.
So, make that option clear in the main requirements text, and then
just cut the whole bullet from the summary.
This further reduces the wall of text on the install instructions.
Simultaneously it lightens up the pressure on this summary to be quite
so terse; expand a couple of items into multiple bullets (yet with
fewer words!) for greater readability.
Now down to just 4 steps!
This version tries to prioritize: just two items that we really want
all admins to look at even if they aren't already mentally committed
to running a big production service and reading all the docs.
Namely, the two required in order to really try out Zulip effectively
with one's colleagues.
The screenshots weren't doing much good without being embedded in the
text... and in fact, looking at them for I think the first time,
they're badly out of date with the app. So cut them.
We might add screenshots later, but on the other hand if we do a good
job with the forms themselves, they should be superfluous.
This further shortens the wall of text inside the instructions.
Note that thanks to embedded reST, we now have the power to provide
custom anchors at section headings! Which in particular means this
link won't break if we later tweak the wording of this heading.
This helps shorten the wall of text between the start of the
instructions and the end. Conversely, now that there are fewer
followup steps, the troubleshooting section at the end isn't so
far away to point.
This flips the experimental `--express` option to be the default.
We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.
The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words. I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.
Also tweak the "production" test suite to match.
This didn't really make much sense - it presented a quickstart summary
of the instructions as an item in the list of differences from the
Ubuntu case, plus that quickstart only incorporated one of the two
actual differences so it didn't work anyway. Given the networking
stuff, an actual quickstart isn't in the cards, so just cut that.
Also fix some small things while we're here.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.
NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.
Fixes: #320 and #291 partially.
Update version of Vagrant from 1.8.6 to 2.0.2.
Update version of VirtualBox from 5.1.8 to 5.2.6.
We needed to update Vagrant because older versions have networking
problems on modern Macs, resulting in weird `apt` errors when
provisioning.
Commit message and some text tweaked by tabbott.
This commit also adds a tool to push translation sources to Transifex.
This tool makes sure that we don't push mobile source file. Mobile
source file is supposed to be handled from Zulip-Mobile repo.
This quiets the last of the warnings that Sphinx was giving us about
documents not being in any toctree, now that we've explicitly told it
with `:orphan:` about the documents we intentionally don't link there.
Now that we have `eval_rst` and can explicitly exclude pages from the
toctree completely, we no longer need to set `includehidden`, and we
can return to using upstream's template.
(Meanwhile, our feature request upstream was successful! See
rtfd/sphinx_rtd_theme#485, which upstream implemented just a week
after we requested it. So that would have been another option.)
This reverts commit 11b8b8f48 "docs: Add rtd layout template."
This is what the Sphinx docs recommend when you actually don't want
the page to be included in navigation:
http://www.sphinx-doc.org/en/stable/markup/toctree.html
And now that we have `eval_rst`, we're able to take advantage of it!
One difference between doing this and the old way of making "hidden"
toctree entries is that with the latter, the "previous" and "next"
links at the bottom of each page would thread through the hidden
entries; which gets kind of confusing when they don't appear in the nav.
The careful testing with dozens of realms was a discrete thing
that happened, not a general state that the tool was in;
certainly not the last state it was known to be in, as the rest
of the sentence explains how it hasn't been so carefully tested
since then. So "as of" doesn't fit.
This is easy now, so make it known to admins who are looking for a
fast path for a test install.
Also totally cut the painfully complicated steps for generating a
self-signed cert by hand. Anyone who actually wants that can find a
hundred explanations on the Web, or can look at our script if they
want to specifically mirror how we do it (which is mercifully much
simpler than this.)
Take the core of the logic from how Debian generates the system's
/etc/ssl/certs/ssl-cert-snakeoil.pem ; that gives me more confidence
in the various config choices, and it also demonstrates a much cleaner
way to use the `openssl` tool. Also replace the outer shell logic for
CLI and logging with a cleaner version.
This allows us to selectively use some of the powerful features of
ReST which Python projects with high-quality documentation (like
Python core, and Django) rely on.
It's now January 2018, so we can delete this caveat, right?
Not quite yet -- the original post we link to now has an
update saying 2018-02-27. Let's make it less specific,
in case the date changes again.
Bulleted information instead of prose, huzzah! Also I think we need
to explain the options a bit right here, or at least link to where
they're documented somewhere. (If the list gets much longer, we'll
want to shift toward the latter.)
Also reorganize existing information a bit, and clean up a couple
of nits.
What I really want is to give these sections nice stable slugs
to put on the anchors and use as the URL fragment, independent of
any wording tweaks on the text headings. But I don't think we
have that feature with Markdown and our current docs infrastructure.
At least for Certbot, the brevity helps make this heading clearer
than the previous one.
Hopefully this version makes it somewhat clearer how the different
methods relate to each other, how to choose between them, what
`ZulipRemoteUserBackend` is for, and how the latter works.
Previous versions of these links were removed earlier today because
Docker upstream had broken them.
To see what the links had been intended to point to, I used the
Wayback Machine (web.archive.org/web), the super-handy project of the
Internet Archive. These are the current equivalents.
We made this change because users often unnecessarily click "Home"
first in their use of Zulip, because it seems appealing. While "All
messages" isn't quite precise (it doesn't include muted streams), it
does describe relatively simply the interleaved view that this
represents.
This commit leaves everything as "home" in the code, and only changes
user-visible strings and docs. Changing the code will be a big project;
there are hundreds of relevant occurrences in variable names, etc.
Further, we'll probably want to convert those various variable names
in different ways.
Tweaked by tabbott to extend the commit message and update a few comments.
Sphinx was displaying "WARNING: document isn't included in any toctree"
for files we just don't want in the TOC. We can hide them from the index,
but the rtd theme defaults to display hidden index entries in the nav bar.
This commit excludes these files from such warnings, and patches layout.html
so that hidden index entries stay hidden from the navigation sidebar.
This commit also moves password-strength.md under docs/production and
adds it as a hidden entry in production/index.rst.
Fixes#7417.
The readthedocs theme overrides a few settings in their layout template.
We might want to change some settings back to their default values.
This commit copies the original readthedocs layout file from
https://github.com/rtfd/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html
to _templates/layout.html, and excludes it from lint and template checks.
Addresses #7417.
This doesn't touch the main path through the install docs; that will
see a broader rewrite soon as we make outbound email optional for a
nascent server, make the hostname and admin email into mandatory
installer flags, and then radically simplify the instructions by
removing mandatory editing of `settings.py` and folding most of what's
left into the installer.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
This adds a few missing entries to the TOC, which hadn't made sense
back when Zulip's ReadTheDocs didn't have the new collapsing feature.
Tweaked by tabbott to also give the SSL certificates doc an
appropriate title for its new role.
Expand bullets "[+]" show up on locals instance, but they
do not show up online on readthedocs. The expand icon is valuable
to indicate that each section has more content.
The problem is readthedocs appends code to conf.py that overrides the
html_theme_options prior to a build. This commit prevents this from
happening so that collapse_navigation remains set to False.
In response to merged PR 7326.
This updates the developer documentation to recommend `git clone -c`
to set the git pull mode to be rebase by default in the repository
configuration.
The hope is that this will help minimize how often new contributors
end up accidentally messing up their PRs by using `git pull`.
Tweaked by tabbott to fix broken links and optimize the language.
Fixes: #7022.
This should get test-documentation and test-help-documentation passing
again after we moved everything around in the last commit.
There are a lot more absolute links (generally in code comments) to
update, but they are lower-priority and can be done in a follow-up
pass.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.
This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.
Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).
Fixes#5265.
The warning here means that the admin can't really act on this yet if
they want to disable email auth, which is likely among admins that
want to make any changes here. And for admins who don't, this is an
extra thing to read and make a decision about before they can get a
server running. See #6985.
Conversely, we already discuss auth backends right at the top of the
`prod-customize` doc, which is linked under "Next steps" at the end of
these instructions.
The warning about EmailAuthBackend is important; but we can move it to
the config file right next to the setting, and then it's available
right when it's actionable, which is if the admin is actually thinking
about changing the setting.
LXC on Debian does not ship with a default network config for
containers, so we now recommend setting up the lxc network config
manually.
Tweaked by tabbott to make these details much harder to miss.
The left navigation bar defaults to "sticking" to the screen as you
scroll. This commit adds the sticky_navigation setting to conf.py to
disable the default behavior.
This addresses part of #5265.
The control panel on the Google side doesn't seem to match the
instructions we have; it looks pretty 2017 to me, so I imagine
it's had a redesign since the instructions were written.
Also, in dev, EXTERNAL_HOST is now a port on zulipdev.com, not on
localhost.
Update these instructions for those developments, and edit lightly.
In dev, recommend setting in `dev_settings` instead of in
`prod_settings_template`; that feels to me a little more reflective of
the actual intent, and the effect should be equivalent.
In some minutes of searching yesterday afternoon on the docs site,
I couldn't find docs on how to set up our OAuth integrations for
the dev environment -- even though I was pretty sure they should
be there somewhere, because I'd just been told that. Wasn't until
I considered the problem fresh today, and grepped the docs source
instead, that I found them.
So, move them to a separate page so they're in the nav.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
The "subdomain" label is redundant, to the extent it's even
accurate -- this is really just the URL we want to display,
which may or may not involve a subdomain. Similarly "external".
The former `external_api_path_subdomain` was never a path -- it's a
host, followed by a path, which together form a scheme-relative URL.
I'm not quite convinced that value is actually the right thing in
2 of the 3 places we use it, but fixing that can start by giving an
accurate name to the thing we have.
This was part of the logic to handle EXTERNAL_API_PATH varying.
But also it was already no longer used -- it was only ever passed
into template contexts, as `external_api_uri`, and it'd been
overtaken there by `external_api_uri_subdomain`.
So, update our dev docs to reflect that, and eliminate the variable.
http://zulip.readthedocs.io/en/latest/architecture-overview.html
mentions "Our mobile clients are separate code repositories: Android
and React Native iOS app. " However Zulip Mobile is the official
mobile Zulip client supporting both iOS and Android.
Edited by tabbott to also fix confusion about the desktop.
The Vagrant environment setup tutorial anchor links for the
Troubleshooting & Common Errors section don't work in GitHub flavored
markdown because the ampersand in the heading gets removed and the
resulting space is filled with a hyphen. (See here:
https://gist.github.com/asabaylus/3071099#gistcomment-1593627)
To make anchor links work with ReadTheDocs as well as with GitHub
markdown for the time being, this commit changes ampersand in the
heading to 'and'.
Fixes#7056.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py
For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful. For configuration and the core
accept/reject logic, it's better to use units that are readily
understood. Switch to those.
I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses. And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft". So, "guesses" it is.
And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting. So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.
(Thanks to @YJDave for a prototype that the code changes in this
commit are based on.)
This has a ton of exclude rules, for two reasons:
(1) We haven't been particularly systematic about avoiding unnecessary
inline style in the past, so there's a lot of code we need to fix.
(2) There are cases where one wants to dynamically compute style
rules. For the latter category, ideally we'd figure out a way to
exclude these automatically (e.g. checking for mustache tags in the
style tag).
We just learned we should be using the "onlytranslated" mode of
Transifex. Since the command is getting a bit complex (and you need
to remember to run `makemessages` first), it makes sense to have a
tool for it.
