We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
Surprisingly hard to find a good base example of a user doc. This one should
at least give the basic pointer to sidebar_index.md and where the new file
goes.
In particular, this improves:
* The explanation of how data is mapped into Zulip
* The explanation of what is printed out by `manage.py query_ldap`
* Makes sure users create their first account with EmailAuthBackend.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond. The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.
Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface. Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username. And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.
It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
There are several situations in which we want to create a Customer and
stripe.Customer object before we really have a billing relationship with a
customer. The main one is giving non-profit or educational discounts.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.
This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).
The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.
Tweaked by tabbott to remove some additional JS code and update the
changelog.
Now that we have nice documentation for our export/import tools, we've
been seeing a lot of users trying to use that as their primary backup
process. Let's correct this.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.
This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.
We include a partial index on UserMessage for this flag.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.
Fixes part of #10042.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.
This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.
For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.
History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org. A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.
Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.
Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
It appears Luke's Dropbox folder/shared link that used to host some
tsearch_extras binaries was removed. It wasn't very high-value
regardless, because most of the platforms involved are deprecated, the
ones that don't generally have a PPA, and building from source is
pretty easy. So, we just remove these options from the documentation.
While we're at it, make clear we only support direct installation on
Ubuntu LTS.
Fixes#9863.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
As detailed in the documentation changes, this simplifies the
development workflow for doing UI work on the /stats pages.
The cost is a ~10% increase the time it takes to run `populate_db`,
which doesn't happen very often (and for most purposes manifests as a
1% increase in the time it takes to rebuild the database from scratch).
This saves about 400ms when running clean-unused-caches, basically by
calling its sub-rountines by import (rather than
`subprocess.check_call()`). The performance optimization seems well worth it.
Fixes#9766.
I don't think this is exactly the right place to document this, but
I'm not sure there's a better one without some restructuring this page
in general (which would probably have value).
Fixes#8769.
I only renamed references that I thought were absolutely necessary
and only if the resulting sentence structure wasn't awkward.
If the renaming resulted in awkward structure, I replaced the term
"webhook" with "integration" (but only in some very obvious cases).
Fixes#9500.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.
We also add a bucket policy, which is part of what one would want to
use this in production.
This addresses much, but not all, of #9361.
We don't reference this anymore (it was only ever used by the Dropbox
integration, which was hardcoded-off for years before being removed in
e6833b6427)
This fixes exceptions when sending PMs in development (where we were
trying to connect to the localhost push bouncer, which we weren't
authorized for, but even if we were, it wouldn't work, since there's
no APNS/GCM certs).
At the same time, we also set and order of operations that ensures one
has the opportunity to adjust the server URL before submitting
anything to us.
This will likely change pretty quickly as we build this feature; I
wrote this just to have a central place to keep track of what we'll
need to document when we ship this feature.
We started doing this for install docs in de2a2d0df, because `latest`
wasn't suitable and because I didn't know about readthedocs's `stable`
feature. The result has been that even with a checklist item, we
don't reliably update the link.
Instead, use the special `stable` version identifier on readthedocs to
link automatically to the highest version it knows about.
This should make it easier to find the templates that are actually
part of the core webapp, instead of having them all mixed together
with the portico pages.
In the very early days of Zulip, we didn't have unread counts; just
the pointer, and the correct behavior when opening a new tab was to
place you near the pointer. That doesn't make any sense now that we
do have unread counts, and this corner case has been a wart for a long
time.
This commit does the main behavior change here. However, there's a
bug we need to fix, where we might end up trying to pre-render a view
of the narrow based on the `all_msg_list` data before `all_msg_list`
is caught up). We need to fix that bug before we can merge this; it
should be possible to determine that using `FetchStatus` on
`all_msg_list`, or with better performance by using the `unread_msgs`
structure to determine whether the message we should be selecting is
present locally.
Fixes#789.
Fixes#9070.
This is a first pass at fixing node docs. This commit eliminates
some text that is either obsolete or just overly confusing, and
it fixes some of the code samples to reflect how the API has
evolved in the last couple years. We also prominently tell
you how to run the tests.
I don't think anybody ever really used this feature, which I
developed but don't even use myself. It kind of runs counter
to the minimalist approach of the rest of node tests.
I would eventually like to re-think the template tests altogether.
They're slow, and we could solve that somewhat by replacing
jsdon/jquery with an HTML parser library to verify structural
things.
It's also possible that we can just rely on our template linters
to catch the biggest class of errors (malformed tags) and let
code review do the rest.
And it's also possible that we should make a second attempt to
ramp up tooling on making it easy to verify templates, but it
doesn't have to be part of the node tests. If we did that, we
would also potentially use tooling for Python-side templates.
These aren't perfect -- in particular "core chat experience" can
probably be broken up -- but I think they help in making a quick skim
work for getting some sense of what the changes are.
This change just reorders and adds headings, with virtually no wording
changes.
This is kind of easy to gloss over, especially with the framing
as a "format"; surely if things work at all, the file format
must have been right, right? It's really a bit more substantive
than that; say so and also add a bit more description.
In addition to many small edits for formatting and clarity, a few more
significant changes:
* In the main instructions, refer specifically to restarting the
server and to testing that the config works.
* Add SendGrid to the recommended list, as it seems like people
give it a somewhat stronger reputation these days than Mailgun.
* Discuss EMAIL_USE_TLS and EMAIL_PORT along with host, user, and
password in the "free services" section. Though those bullets feel
kind of duplicative to me already.
Let's get right to the point of how to configure SMTP once you know
what you want. That section is pretty short anyway; and we can have
a first step direct the reader to our suggestions if they don't know
what service they want to use.
Also adjust the hierarchy of the headings: group the various
alternative email services under one heading, and group
troubleshooting together under an independent heading.
Also correct what we say about EMAIL_PORT: the Django default is
apparently 25, so if the provider *does* use the usual port 587
then we'll need the port to be set.
In our new system for updating realm settings, we don't need to create
separate functions to update the input elements for each feature.
Most of the work is done automatically by
`settings_org.sync_realm_settings`.
Usually, to debug a small change, you have to remove some tests from JSON
because of lack of support for comments in JSON. This commit allows to
ignore some tests by setting `"ignore" : true` in the bugdown fixtures.
Also, since this is only for while developing, the complete test suite will
throw an error if we leave an 'ignored' test in a commit.
This will allow realm admins to remove others from private stream to
which the realm administrator is not subscribed; this is important for
managing those streams, because previously nobody could remove users
from private streams that didn't have any realm administrators
subscribed.
This will allow realm admins to access subscribers of unsubscribed
private stream. This is a preparatory commit for letting realm admins
remove those users.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
The main point of this change is to streamline the core
code for input pills, and we use also modify user groups.
The main change to input_pill.js is that you now
configure a function called `create_item_from_text`, and
that can return an arbitrary object, and it just needs
a field called `display_value`.
Other changes:
* You now call `input.create(opts)` to create the
widget.
* There is no longer a cache, because we can
write smarter code in typeahead `source` functions
that exclude ids up front.
* There is no value/optinalKey complexity, because
the calling code can supply arbitrary objects and
do their own external data management on the pill
items.
* We eliminate `prependPill`.
* We eliminate `data`, `keys`, and `values`, and just
have `items`.
The summary already has a qualifier that basically says it shouldn't
matter for most people -- making it simultaneously the most
complicated bullet there, and among the least likely to matter.
And in fact, this requirement shouldn't matter for *anyone* when first
experimenting with Zulip. If certbot won't work in a given admin's
environment, and the available ways to get a cert aren't convenient,
they can always let the installer generate a self-signed cert to get
going, and circle back to the issue later.
So, make that option clear in the main requirements text, and then
just cut the whole bullet from the summary.
This further reduces the wall of text on the install instructions.
Simultaneously it lightens up the pressure on this summary to be quite
so terse; expand a couple of items into multiple bullets (yet with
fewer words!) for greater readability.
Now down to just 4 steps!
This version tries to prioritize: just two items that we really want
all admins to look at even if they aren't already mentally committed
to running a big production service and reading all the docs.
Namely, the two required in order to really try out Zulip effectively
with one's colleagues.
The screenshots weren't doing much good without being embedded in the
text... and in fact, looking at them for I think the first time,
they're badly out of date with the app. So cut them.
We might add screenshots later, but on the other hand if we do a good
job with the forms themselves, they should be superfluous.
This further shortens the wall of text inside the instructions.
Note that thanks to embedded reST, we now have the power to provide
custom anchors at section headings! Which in particular means this
link won't break if we later tweak the wording of this heading.
This helps shorten the wall of text between the start of the
instructions and the end. Conversely, now that there are fewer
followup steps, the troubleshooting section at the end isn't so
far away to point.
This flips the experimental `--express` option to be the default.
We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.
The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words. I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.
Also tweak the "production" test suite to match.
This didn't really make much sense - it presented a quickstart summary
of the instructions as an item in the list of differences from the
Ubuntu case, plus that quickstart only incorporated one of the two
actual differences so it didn't work anyway. Given the networking
stuff, an actual quickstart isn't in the cards, so just cut that.
Also fix some small things while we're here.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.
NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.
Fixes: #320 and #291 partially.
Update version of Vagrant from 1.8.6 to 2.0.2.
Update version of VirtualBox from 5.1.8 to 5.2.6.
We needed to update Vagrant because older versions have networking
problems on modern Macs, resulting in weird `apt` errors when
provisioning.
Commit message and some text tweaked by tabbott.
This commit also adds a tool to push translation sources to Transifex.
This tool makes sure that we don't push mobile source file. Mobile
source file is supposed to be handled from Zulip-Mobile repo.
This quiets the last of the warnings that Sphinx was giving us about
documents not being in any toctree, now that we've explicitly told it
with `:orphan:` about the documents we intentionally don't link there.
Now that we have `eval_rst` and can explicitly exclude pages from the
toctree completely, we no longer need to set `includehidden`, and we
can return to using upstream's template.
(Meanwhile, our feature request upstream was successful! See
rtfd/sphinx_rtd_theme#485, which upstream implemented just a week
after we requested it. So that would have been another option.)
This reverts commit 11b8b8f48 "docs: Add rtd layout template."
This is what the Sphinx docs recommend when you actually don't want
the page to be included in navigation:
http://www.sphinx-doc.org/en/stable/markup/toctree.html
And now that we have `eval_rst`, we're able to take advantage of it!
One difference between doing this and the old way of making "hidden"
toctree entries is that with the latter, the "previous" and "next"
links at the bottom of each page would thread through the hidden
entries; which gets kind of confusing when they don't appear in the nav.
The careful testing with dozens of realms was a discrete thing
that happened, not a general state that the tool was in;
certainly not the last state it was known to be in, as the rest
of the sentence explains how it hasn't been so carefully tested
since then. So "as of" doesn't fit.
This is easy now, so make it known to admins who are looking for a
fast path for a test install.
Also totally cut the painfully complicated steps for generating a
self-signed cert by hand. Anyone who actually wants that can find a
hundred explanations on the Web, or can look at our script if they
want to specifically mirror how we do it (which is mercifully much
simpler than this.)
Take the core of the logic from how Debian generates the system's
/etc/ssl/certs/ssl-cert-snakeoil.pem ; that gives me more confidence
in the various config choices, and it also demonstrates a much cleaner
way to use the `openssl` tool. Also replace the outer shell logic for
CLI and logging with a cleaner version.
This allows us to selectively use some of the powerful features of
ReST which Python projects with high-quality documentation (like
Python core, and Django) rely on.
It's now January 2018, so we can delete this caveat, right?
Not quite yet -- the original post we link to now has an
update saying 2018-02-27. Let's make it less specific,
in case the date changes again.
Bulleted information instead of prose, huzzah! Also I think we need
to explain the options a bit right here, or at least link to where
they're documented somewhere. (If the list gets much longer, we'll
want to shift toward the latter.)
Also reorganize existing information a bit, and clean up a couple
of nits.
What I really want is to give these sections nice stable slugs
to put on the anchors and use as the URL fragment, independent of
any wording tweaks on the text headings. But I don't think we
have that feature with Markdown and our current docs infrastructure.
At least for Certbot, the brevity helps make this heading clearer
than the previous one.
Hopefully this version makes it somewhat clearer how the different
methods relate to each other, how to choose between them, what
`ZulipRemoteUserBackend` is for, and how the latter works.
Previous versions of these links were removed earlier today because
Docker upstream had broken them.
To see what the links had been intended to point to, I used the
Wayback Machine (web.archive.org/web), the super-handy project of the
Internet Archive. These are the current equivalents.
We made this change because users often unnecessarily click "Home"
first in their use of Zulip, because it seems appealing. While "All
messages" isn't quite precise (it doesn't include muted streams), it
does describe relatively simply the interleaved view that this
represents.
This commit leaves everything as "home" in the code, and only changes
user-visible strings and docs. Changing the code will be a big project;
there are hundreds of relevant occurrences in variable names, etc.
Further, we'll probably want to convert those various variable names
in different ways.
Tweaked by tabbott to extend the commit message and update a few comments.
Sphinx was displaying "WARNING: document isn't included in any toctree"
for files we just don't want in the TOC. We can hide them from the index,
but the rtd theme defaults to display hidden index entries in the nav bar.
This commit excludes these files from such warnings, and patches layout.html
so that hidden index entries stay hidden from the navigation sidebar.
This commit also moves password-strength.md under docs/production and
adds it as a hidden entry in production/index.rst.
Fixes#7417.
The readthedocs theme overrides a few settings in their layout template.
We might want to change some settings back to their default values.
This commit copies the original readthedocs layout file from
https://github.com/rtfd/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html
to _templates/layout.html, and excludes it from lint and template checks.
Addresses #7417.
This doesn't touch the main path through the install docs; that will
see a broader rewrite soon as we make outbound email optional for a
nascent server, make the hostname and admin email into mandatory
installer flags, and then radically simplify the instructions by
removing mandatory editing of `settings.py` and folding most of what's
left into the installer.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
This adds a few missing entries to the TOC, which hadn't made sense
back when Zulip's ReadTheDocs didn't have the new collapsing feature.
Tweaked by tabbott to also give the SSL certificates doc an
appropriate title for its new role.
Expand bullets "[+]" show up on locals instance, but they
do not show up online on readthedocs. The expand icon is valuable
to indicate that each section has more content.
The problem is readthedocs appends code to conf.py that overrides the
html_theme_options prior to a build. This commit prevents this from
happening so that collapse_navigation remains set to False.
In response to merged PR 7326.
This updates the developer documentation to recommend `git clone -c`
to set the git pull mode to be rebase by default in the repository
configuration.
The hope is that this will help minimize how often new contributors
end up accidentally messing up their PRs by using `git pull`.
Tweaked by tabbott to fix broken links and optimize the language.
Fixes: #7022.
This should get test-documentation and test-help-documentation passing
again after we moved everything around in the last commit.
There are a lot more absolute links (generally in code comments) to
update, but they are lower-priority and can be done in a follow-up
pass.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.
This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.
Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).
Fixes#5265.
The warning here means that the admin can't really act on this yet if
they want to disable email auth, which is likely among admins that
want to make any changes here. And for admins who don't, this is an
extra thing to read and make a decision about before they can get a
server running. See #6985.
Conversely, we already discuss auth backends right at the top of the
`prod-customize` doc, which is linked under "Next steps" at the end of
these instructions.
The warning about EmailAuthBackend is important; but we can move it to
the config file right next to the setting, and then it's available
right when it's actionable, which is if the admin is actually thinking
about changing the setting.
LXC on Debian does not ship with a default network config for
containers, so we now recommend setting up the lxc network config
manually.
Tweaked by tabbott to make these details much harder to miss.
The left navigation bar defaults to "sticking" to the screen as you
scroll. This commit adds the sticky_navigation setting to conf.py to
disable the default behavior.
This addresses part of #5265.
The control panel on the Google side doesn't seem to match the
instructions we have; it looks pretty 2017 to me, so I imagine
it's had a redesign since the instructions were written.
Also, in dev, EXTERNAL_HOST is now a port on zulipdev.com, not on
localhost.
Update these instructions for those developments, and edit lightly.
In dev, recommend setting in `dev_settings` instead of in
`prod_settings_template`; that feels to me a little more reflective of
the actual intent, and the effect should be equivalent.
In some minutes of searching yesterday afternoon on the docs site,
I couldn't find docs on how to set up our OAuth integrations for
the dev environment -- even though I was pretty sure they should
be there somewhere, because I'd just been told that. Wasn't until
I considered the problem fresh today, and grepped the docs source
instead, that I found them.
So, move them to a separate page so they're in the nav.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
The "subdomain" label is redundant, to the extent it's even
accurate -- this is really just the URL we want to display,
which may or may not involve a subdomain. Similarly "external".
The former `external_api_path_subdomain` was never a path -- it's a
host, followed by a path, which together form a scheme-relative URL.
I'm not quite convinced that value is actually the right thing in
2 of the 3 places we use it, but fixing that can start by giving an
accurate name to the thing we have.
This was part of the logic to handle EXTERNAL_API_PATH varying.
But also it was already no longer used -- it was only ever passed
into template contexts, as `external_api_uri`, and it'd been
overtaken there by `external_api_uri_subdomain`.
So, update our dev docs to reflect that, and eliminate the variable.
http://zulip.readthedocs.io/en/latest/architecture-overview.html
mentions "Our mobile clients are separate code repositories: Android
and React Native iOS app. " However Zulip Mobile is the official
mobile Zulip client supporting both iOS and Android.
Edited by tabbott to also fix confusion about the desktop.
The Vagrant environment setup tutorial anchor links for the
Troubleshooting & Common Errors section don't work in GitHub flavored
markdown because the ampersand in the heading gets removed and the
resulting space is filled with a hyphen. (See here:
https://gist.github.com/asabaylus/3071099#gistcomment-1593627)
To make anchor links work with ReadTheDocs as well as with GitHub
markdown for the time being, this commit changes ampersand in the
heading to 'and'.
Fixes#7056.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py
For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful. For configuration and the core
accept/reject logic, it's better to use units that are readily
understood. Switch to those.
I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses. And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft". So, "guesses" it is.
And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting. So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.
(Thanks to @YJDave for a prototype that the code changes in this
commit are based on.)
This has a ton of exclude rules, for two reasons:
(1) We haven't been particularly systematic about avoiding unnecessary
inline style in the past, so there's a lot of code we need to fix.
(2) There are cases where one wants to dynamically compute style
rules. For the latter category, ideally we'd figure out a way to
exclude these automatically (e.g. checking for mustache tags in the
style tag).
We just learned we should be using the "onlytranslated" mode of
Transifex. Since the command is getting a bit complex (and you need
to remember to run `makemessages` first), it makes sense to have a
tool for it.
This commit updates various places where check_send_message had
been previously recommended to recommend check_send_stream_message
for sending messages to a public stream.
This accumulates several changes in recent commits: decimal point
rather than comma, compact log level, and logger names, the latter
abbreviated `zr` in the case of `zulip.request`.
Collaboration in area label teams is only available to members of the
Zulip organization on GitHub. For non-members the related links are
not working, which can be confusing. Address this by explaining the
links won't work and also that anyone can join.
This is just enough of a quick fix to work with a stock Zulip 1.6
server. We should really also make this robust to arbitrary input
from the remote Zulip server, even though it'll be a little tedious.
This is a start and fixes the most glaring problems from not updating
this documentation; I'd like Harshit to do a helpful pass on updating
this to cover some of the more subtle details about how our emoji
picker works, emoji aliases, etc.
This change means that almost every Zulip server out there will now be
using subdomains for every realm. There are a few complications noted
in the release notes.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.
This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
Since we've found that it's fairly frequent that we want to recommend
to developers that they upgrade to a version of Zulip from Git, it
makes sense to include that by default.
This keeps the examples in line with our actual codebase.
Also while I'm here revise it to explain the actual motivation for our
use of `env`, and to correct some subtle details -- it's actually the
kernel that interprets the shebang (as visible in e.g. a `strace` log),
not the shell, and when the program is executed as `./my_program.py`
the exact name including `./` is passed to the interpreter.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2. In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.
One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2. See discussion on the respective previous
commits that made those explicit. There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
This works fine in my testing -- I followed it on a fresh `trusty` VM,
after just getting SSL certs with our LetsEncrypt instructions, and
the install completed successfully.
And in the source tree, the only evidence I can find of a potential
remaining dependency on the `/root/zulip` path is the Nagios config in
`puppet/zulip_ops/`. That's actually already broken, in that it
depends on `/root/zulip/api/`, so we'll have to sort that out;
and in any case, it doesn't matter to most people installing Zulip.
Add a mention that the install script will move the just-unpacked
directory out from under you. While we're here, add a few words about
where the deployed code is laid out.
Text of those last words tweaked by tabbott.
This follows up on 207cf6302 from last year to clean up cases that
have apparently popped up since then. Invoking the scripts directly
makes a cleaner command line in any case, and moreover is essential
to how we control running a Zulip install as either Python 2 or 3
(soon, how we always ensure it runs as Python 3.)
One exception: we're currently forcing `provision` in dev to run
Python 3, while still running both Python 2 and Python 3 jobs in CI.
We use a non-shebang invocation to do the forcing of Python 3.
Since we're auto-detecting the value anyway, there's no reason it
can't be moved to DEFAULT_SETTINGS.
This lets us remove some clutter from the installation documentation.
This field is convenient for bankruptcy checks. Clients could
calculate it from page_params.unread_msgs before this change, but
it would kind of a painful calculation.
To add count, we had to simplify the mypy annotations, which weren't
really accurate before.
Update Email, Beanstalk, Hubot, JIRA, and Trello integrations
links.
The Hubot integrations section (/integrations#hubot-integrations)
was removed in an earlier redesign of /integrations. This commit
replaces the link with the hubot-scripts organization on
Github, which displays the comprehensive list of all integrations
available via Hubot.
Fixes#5875.
In some of these contexts, we may still be *using* the Python 2
version, but at least this should eliminate running into
`ImportError`s one by one in scripts that run outside a virtualenv,
as we update their shebangs to refer to Python 3.
Several Python libraries we use don't come in Python 3 versions on
trusty: gevent, boto, twisted, django, django-tagging, whisper.
The latter two don't come in Python 3 versions even on xenial.
So some work required before we can actually switch the code that
relies on those libraries to run as Python 3 -- probably the best
solution will be to backport them all in our apt repo. (All but
`whisper` are packaged in zesty; `whisper` upstream just grew Python 3
support this year.)
We are phasing out the following in tests:
add_dependencies - this is just kind of a clunky UI
require - normal JS requires cause test leaks
In order to plug require leaks, we are effectively doing what
we always have done inside of add_dependencies, which is to
keep track of which modules we have done `require` on, and
these get cleared between tests.
Now we just use `zrequire` every time we want to pull in real
code to our global namespace.
I think soon we'll put the Python 3 venv at `/srv/zulip-venv` and
make `/srv/zulip-py3-venv` just an alias to that (then remove the
alias too), but for now the old name is helpful for spotting places
that need an update.
The `/srv/zulip-venv` name still appears in codepaths used by Travis
tests.
From the line in tools/provision it should trickle to the rest of the
scripts. This works since almost all the python scripts have been linted
to be generic.
Proof that the setup is python3 only: with this commit, within the
vagrant container env, /srv/zulip-venv is no longer present and
`./tools/run-dev.py` runs just fine.
[gnprice: Added `rm -f` and warning message, and made small edits.]
This makes supervisor see the service as cheerfully running
and let it alone, rather than constantly retry starting it.
Because the crash/restart loop means repeatedly spending a
couple of seconds loading Django and the app, separated by
brief periods while supervisor notices the crash and acts
on it, it was actually consuming about 30-50% CPU on the
zulipchat.com staging server.
This pulls together what's covered in detail in several
longer pages, and gives us a page that can serve as a good
drop-in replacement for https://zulip.org/server.html .
Also tweak a couple of related bits for clarity and orthography.
We are adding a new list of unread message ids grouped by
conversation to the queue registration result. This will allow
clients to show accurate unread badges without needing to load an
unbound number of historic messages.
Jason started this commit, and then Steve Howell finished it.
We only identify conversations using stream_id/user_id info;
we may need a subsequent version that includes things like
stream names and user emails/names for API clients that don't
have data structures to map ids -> attributes.
After editing this file for the previous commit, I couldn't unsee
"low-traffic the". So I fixed that, saw more issues, and then just
went through the page for everything that was similarly trivial to fix.
This allow the webbpack dev server to properly reload JavaScript modules
while running in dev without restarting the server. We need to connect
to webpack-dev-server directly because SockJS doesn't support more than
one connection on the same host/port.
ScheduledJob was written for much more generality than it ended up being
used for. Currently it is used by send_future_email, and nothing
else. Tailoring the model to emails in particular will make it easier to do
things like selectively clear emails when people unsubscribe from particular
email types, or seamlessly handle using the same email on multiple realms.
This new setting controls whether or not users are allowed to see the
edit history in a Zulip organization. It controls access through 2
key mechanisms:
* For long-ago edited messages, get_messages removes the edit history
content from messages it sends to clients.
* For newly edited messages, clients are responsible for checking the
setting and not saving the edit history data. Since the webapp was
the only client displaying it before this change, this just required
some changes in message_events.js.
Significantly modified by tabbott to fix some logic bugs and add a
test.
This commit does the following things:
* Instead of using a manual tool for downloading sprite sheets, use
`emoji-datasource` npm package.
* Modify the `build_emoji` script to use sprite sheets from the npm
package.
Bumps PROVISION_VERSION.
Fixes: #4730.
NPM packages should be installed at the beginning of the provisioning
process so that later in the provisioning process if a script requires
any NPM package it can use it. Earlier, we were installing NPM packages
in the last as the installation process can fail due to network issues
but since we now retry in case the installation fails, they can be
installed safely at the beginning of the process as well just like apt
packages.
In this commit we basically start to override the request method of
httplib2.Http() to raise an exception whenever it is called i.e.
a trial is made to access the network from test suits.
Fixes: #1472.
This old third party library added support
for a "mousewheel" event to detect scrolling.
However, it is not compatible with jQuery 3
and is obsolete now that there is a standard
"wheel" event that accomplishes the same thing.
I found the new paragraph from 74d83cc47 somewhat hard to follow,
so here's another version. Also try to make the structure of
the rest of the section around it somewhat more clear.
This commit adds a new linter which runs from tools/travis/backend.
It runs over the translations.json file and checks if any of the
translatable string contains handlebars in it.
Fixes#5544
This started as a PSA in the form of a series of chat messages in
`#general` on chat.zulip.org; putting them here, with some editing,
to make their value more durable.
Also rearrange this doc slightly so that it's not specific to
the server codebase, except in a few explicit spots.
The bit that's for authors should probably be somewhere else.
I think there isn't right now a great natural spot for it --
probably the top of docs/git-guide, some parts of docs/version-control,
and that paragraph here should all turn into a top-level "guide
to submitting code to Zulip" doc, which would link to the rest
of docs/git-guide and to some other resources. Leaving that
for another day.
This follows up the recent commit
3d1d09b3d docs: Remove discussion of old Django templating engine.
with a small grammar fix and removing another vestige of making the
distinction between Jinja2 and Django templates, and also rearranges
the logic slightly to reflect that backend and frontend templates
have separate sections.
Probably a bigger restructuring is in order to help the reader
navigate through all the good content in this doc, but that's
a bigger job for another day.
- Remove `perfect-scrollbar` from `static/third` and fetch it from npm.
- Upgrade `perfect-scrollbar` to 0.7.1.
- Bump up the `PROVISION_VERSION` to 5.6.
Changed `wheelSpeed` in "static/js/scroll_bar.js" to 0.5, because when it
20, the scrollbar scrolls very fast.
Changed 'wheelSpeed' in "static/js/emoji_picker.js" from 25 to 0.68
(based on tabbott's testing of scrolling through the emoji list).
Part of #1709.
Introduce Swagger UI and the Swagger/OpenAPI specification. Explain
the structure of zulip.yaml and show examples of different sections
of the file.
This is a new file in /docs not yet included in the Read the Docs
table of contents. Where it should go should be determined as we iterate
on the Swagger UI integration and expand REST API doc coverage using it.
For more on Swagger UI and how Zulip uses it, see:
https://github.com/zulip/zulip/issues/3474https://github.com/zulip/zulip/pull/3397
With some minor tweaks to advertise this by tabbott.
The formatting was coming out pretty badly on readthedocs,
and also I think the exposition could be a little clearer.
I still don't love how the formatting comes out, but
I think this is pretty OK.
There were 2 things wrong here:
(1) The new emoji cache directories weren't being created properly
(2) We weren't downloading the new emoji sprite sheets.
I think based on this experience, we should definitely invest in
moving more platforms to use provision.py.
Fixes#5160.
I faced this problem many a times, might be of help to
beginners. Because, the same thing doesn't work when done through
`vagrant suspend` followed by `vagrant up`.
Due to the directory symlink structure in the dev VM, including the
`~/zulip/contrib_bots/` prefix in the command for running a bot causes
`run.py` to fail with an error.
Aka the current "testing" release, expected to graduate to "stable"
later in 2017.
Fortunately the instructions are very similar to those for
Ubuntu 16.04 and 14.04 -- two packages don't exist, and
those two packages turn out (empirically, on my laptop)
not to be necessary.
Leave most references to "Ubuntu" still just saying "Ubuntu",
on the theory that Debian users will generally follow those
breadcrumbs where they lead and in order to keep lists short.
The integration-guide has now been updated to reflect the recent
decision to rename webhook fixtures from
<webhook_name>/fixtures/<webhook_name>_<event_type>.json to
<webhook_name>/fixtures/<event_type>.json.
The webhook-walkthrough has now been updated to reflect the recent
decision to rename webhook fixtures from
<webhook_name>/fixtures/<webhook_name>_<event_type>.json to
<webhook_name>/fixtures/<event_type>.json.
This commit changes the backend testing framework to run
in parallel mode which is same as --processes=4. If --coverage
is supplied, we enforce serial mode, --processes=1, because
coverage is not compatible with parallel mode at the moment.
Previously, api_key_only_webhook_view passed 3 positional arguments
(request, user_profile, and client) into a function. However, most
of our other auth decorators only pass 2 positional arguments. For
the sake of consistency, we now make api_key_only_webhook_view set
request.client and pass only request and user_profile as positional
arguments.
Modified composebox_typeahead.js to recognize the triple backtick
and tilde for code blocks, and added appropriate typeahead functions
in that file and in typeahead_helper.js.
Additionally, a new file pygments_data.js contains a dictionary of
the supported languages, mapping to relative popularity
rankings. These rankings determine the order of sort of the
languages in the typeahead.
This JavaScript file is actually in static/generated/pygments_data.js, as it
is generated by a Python script, tools/build_pymgents_data.py. This is
so that if Pygments adds support for new languages, the JavaScript file
will be updated appropriately. This python script uses a set of popularity
rankings defined in lang.json.
Corresponding unit tests were also added.
Fixes#4111.
The integration guide has now been updated to reflect the recent
decision to store webhook fixtures in
zerver/webhooks/<webhook_name>/fixtures/ as opposed to
zerver/fixtures/<webhook_name>/.
The webhook walkthrough has now been updated to reflect the recent
decision to store webhook fixtures in
zerver/webhooks/<webhook_name>/fixtures/ as opposed to
zerver/fixtures/<webhook_name>/.
Update the steps listed in the "Writing a new application
feature" section, based on changes to how realm properties
are created and updated on the backend.
Addresses part of issue #3854.
Rename 'zulip_internal' decorator to 'require_server_admin', add
documentation for 'server_admin', explaining how to give permission
for ./activity page.
Fixes: #1463.
Tweaked this warning because we currently check if there are any
uncommited changes before we reset to a specific pull request.
Changes expanded a bit by tabbott.
It turned out to not be that useful once we added subgroup. The previous
design of the CountStat object also assumed more reuseability of the *_query
strings than what ended up happening.
The filter_args also had some carrying costs:
* It's hard to be confident that filter_args other than the ones explicitly
in our tests would have had expected behavior.
* The filter_args/join_args system is the most complex part of the CountStat
object, and makes understanding the *_query strings unnecessarily
difficult for a new contributor.
Groundwork for allowing stats like "Monthly Active Users".
CountStat.interval is no longer as clean a value as before, so removed it
from views.get_chart_data. It wasn't being used by the frontend anyway.
Removing interval from logger calls in counts.py is not a big loss since we
now include the frequency (which is typically also the interval) in
CountStat.property.
This adds the option '--rerun' to the `test-backend` infrastructure.
It runs the tests that failed during the last 'test-backend' run. It
works by stailing failed test info at var/last_test_failure.json
Cleaned up by Umair Khan and Tim Abbott.
This better sets expectatations for the fact that in Zulip, the
Organization settings UI is available read-only to non-administrator
users.
Tweaked by tabbott to update some additional references.
This is mostly a straight port from bash to Python, but we
rename the coverage option to `--coverage` and we add checks
for being in a venv and being correctly provisioned.
Fixes#4009.
While it's sometimes nice to put a few selectors on the same line,
it is generally better to have a consistent way of formatting our
selectors, and most of our code up until now lists them vertically.
This change fixes the linter to enforce one selector per line, and
it cleans up the places in the CSS where we had multiple selectors
on the same line.
The advantages of one-per-line are as followers:
* cleaner diffs
* easier to see when multiple areas of the app may have the
same format
* less likely to go over 80 cols
* makes it more clear where we have deep nesting in the
individual selectors
* makes it easier for our linting tools to enforce
whitespace violations
This also fixed an old bug where we had ".landing_page h2, h4", which
sets "h4" styles outside of the landing page.
This changes the layout of administration for non-administrators such
that they can view organization settings and emoji settings and
displays everything as readonly unless they have the capability to edit.
For now, we just enabled this for the emoji settings and organization
settings features.
This covers the standard multi-step process for doing large
migrations, as well as other important properties to consider when
writing migrations.
Also documents the new Django 'atomic=False' option.
Fixes#1332.
Our lists of rabbitmq queues was likely to end up out of date, since
there was nothing enforcing that the various lists of queues were
correct or the same as each other.
Add a new section after the Hello World walkthrough for additional detail that
doesn't directly apply to this example. Included are discussions on creating
negative tests and handling custom query parameters.
Remaining integration of the material originally for #3478
Fill in additonal detail following the existing document structure.
Includes authentication, custom streams, negative tests, and types
of test data and fixtures. Also fix typos and reformat to match the
new integration doc style.
Partial integration of the material originally for #3478
This adds an assertion, when `test-backend` is run with `--coverage`,
that we have 100% test coverage on a list of files that we expect to.
There's a whitelist/blacklist, managed in tools/test-backend.
Fixes#3363.
This moves do_events_register, fetch_initial_state_data and friends to
a new file.
Modified significantly by tabbott for correctness and to remove unused
imports.
Fixes#3635.
Having just found that a number of our casper tests were buggy because
they were using casper.waitForSelector and fixed those, this seemed
like a good opportunity to update the docs to recommend the selectors
that more faithfully do what developers expect them to.
While I was at it, I tried to make this a bit better organized, though
I think more work could be done on that front.
Breaks out the Hello World example to create a new
webhook-walkthrough.md. Includes minor edits so the two docs
read well. Adds the new page, "Webhook walkthrough", to the TOC.
Fixes#3498
Apparently, the updated version of this has a serious scrolling
performance problem in the left sidebar that basically makes scrolling
in that area unusable.
This reverts commit b683b2d3c3.
- Remove `jquery-mousewheel` from `static/third` and fetch it from npm.
- Upgrade `jquery-mousewheel` to 3.1.6.
- Bump up the `PROVISION_VERSION` to 4.5.
- Change some js code to comply with this `jquery-mousewheel` version.
Part of #1709.
- Remove `underscore.js` from `static/third` and fetch it from `npm`.
- Upgrade `underscore.js` to 1.8.3.
- Bump up the `PROVISION_VERSION` to 4.2.
Part of #1709
- Remove `codepointat` from `static/third` and fetch it from `npm`.
- Upgrade `codepointat` to 0.2.0.
- Bump up the `PROVISION_VERSION` to 4.1.
Part of #1709.
- Remove `winchan.js` from `static/third` and fetch it from `npm`.
- Upgrade `winchan` to 0.2.0.
- Bump up the `PROVISION_VERSION` to 4.0.
Part of #1709.
Before this commit, provisioning was done by executing provision.py,
which printed the log directly to stdout, making debugging harder.
This commit creates a wrapper bash script 'provision' in tools, which
calls 'zulip/scripts/tools/provision_vm.py' (the new location of
provision.py) and prints all the output to
'zulip/var/log/zulip/zulip_provision.log' via 'tee'.
Travis tests and docs have been modified accordingly.
This reverts commit 7bf10ec74f.
Apparently, SockJS 1.1.1 is broken with the browser used in our legacy
desktop app, resulting in messages being silently not sent.
This adds some configuration options to settings.py, namely
PASSWORD_MIN_LENGTH and PASSWORD_MIN_QUALITY, which control
when the frontend validator invalidates the password.
Closes#2628
- Replace download-zxcvbn with downloading it from npm.
- Change zxcvbn.js path to node_modules (because npm put it to
`node_modules` directory.
- Bump `PROVISION_VERSION` in `version.py` to 2.4.
Fixes#2423.
The previous RTD headings were unclear and resulted in some users
attempting to install the Zulip production environment for doing
development. This clarifies the situation.
Based on feedback from first-time contributors, this commit simplifies
the number of paths available for finding the dev setup directions you
need. Setup instructions are now organized into Recommended (Vagrant)
and Advanced (non-Vagrant).
In order to get the ToC to display correctly, I've combined all the
advanced, non-Vagrant methods into one page. I've left the old pages
intact, with a note that the content has been moved and link to the new
page. This is in case folks have linked directly to those pages.
The advanced setup directions still need to be cleaned up, but this is a
start.
This reorders the sections such that setting up the development
environment and making sure it runs comes before editing the
code. Also added some clarifying sentences to give context to bullets.
Two reasons not to use such links:
- when making doc changes, if you follow links in your local build, they can
cause you to silently end up no longer reading your local changes
- they can cause you to randomly switch between http:// and https://
(This is mostly an internal dev document for now. We should eventually
link it into our RTD system, perhaps after we think we are mostly
bug-free in terms of what the doc specifies.)
