-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
user_groups: Add markdown rendering for group descriptions. #32873
base: main
Are you sure you want to change the base?
Conversation
Hey @alya , this PR is ready for review. Could you please take a look when you have a moment? Thanks! |
The screenshots look great! Please resolve the conflicts to prepare this for review. |
@alya done, please check! |
@sahil839 could you please take a look at this one? I haven't tested. |
zerver/openapi/zulip.yaml
Outdated
displaying this content so that emoji, LaTeX, and other syntax | ||
work correctly. And any client-side security logic for | ||
user-generated message content should be applied when displaying | ||
this HTML as though it were the body of a Zulip message. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This and other changes in docs needs a "Changes" entry. Also, changelog.md
and version.py
should be updated accordingly.
zerver/lib/user_groups.py
Outdated
} | ||
|
||
full_members_group_info = { | ||
"name": SystemGroups.FULL_MEMBERS, | ||
"description": "Members of this organization, not including new accounts and guests", | ||
"rendered_description": "<p>Members of this organization, not including new accounts and guests</p>", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of doing this, can we just use render_group_description
when creating the system groups?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used render_group_description
instead but, there was circular import issue, so I tried importing from zerver.actions.user_groups import render_group_description
in function, it seems to work. can you please confirm if it is permitted?
web/src/typeahead_helper.ts
Outdated
render_typeahead_item({ | ||
primary: user_groups.get_display_group_name(user_group.name), | ||
secondary: user_group.description, | ||
// eslint-disable-next-line unicorn/prefer-string-replace-all | ||
secondary_html: user_group.rendered_description.replace(/^<p>|<\/p>$/g, ""), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does this need replace
but we do not do the same for streams?
9b21d22
to
74a0f01
Compare
Hello @sahil839, I have made the changes, sorry I was busy for a few days. can you please take a look again, when you get time. thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Posted a couple of comments. Can do a complete review along with manual testing once you rebase the PR to resolve merge conflicts.
@@ -899,6 +902,13 @@ def create_system_user_groups_for_realm(realm: Realm) -> dict[str, NamedUserGrou | |||
everyone_on_internet_group_info, | |||
] | |||
|
|||
from zerver.actions.user_groups import render_group_description |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we just define render_group_description
in this file itself, ie. zerver/lib/user_groups.py
if that doesn't result in any import cycles?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from zerver.lib.markdown import markdown_convert
ImportError: cannot import name 'markdown_convert' from partially initialized module 'zerver.lib.markdown' (most likely due to a circular import) (/srv/zulip/zerver/lib/markdown/__init__.py)
from zerver.lib.mention import render_group_description
ImportError: cannot import name 'render_group_description' from partially initialized module 'zerver.lib.mention' (most likely due to a circular import) (/srv/zulip/zerver/lib/mention.py)
ERROR:root:Failed to forward GET /json/events to port 9993: Cannot connect to host 127.0.0.1:9993 ssl:default
....
from zerver.lib.user_groups import (
ImportError: cannot import name 'get_group_setting_value_for_api' from partially initialized module 'zerver.lib.user_groups' (most likely due to a circular import) (/srv/zulip/zerver/lib/user_groups.py)
....
from zerver.lib.user_groups import (
ImportError: cannot import name 'get_group_setting_value_for_api' from partially initialized module 'zerver.lib.user_groups' (most likely due to a circular import) #(/srv/zulip/zerver/lib/user_groups.py)
tried this! still encountering circular import error, also tried modifying other related files too, haven't got anywhere.
It'll be great if you can find time to check it out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this seems fine for now then.
zerver/openapi/zulip.yaml
Outdated
rendered_description: | ||
type: string | ||
description: | | ||
The description of the user group rendered as HTML. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You would need to add **Changes**
entry for all these added blocks.
@sahil839 , made the changes, can you please take a look again. thanks! |
zerver/openapi/zulip.yaml
Outdated
user-generated message content should be applied when displaying | ||
this HTML as though it were the body of a Zulip message. | ||
|
||
**Changes**: Prior to Zulip 10.0 (feature level 338), markdown rendering |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think "New in Zulip 10.0 (feature level 338), for adding support for markdown rendering of group descriptions." would be a better phrasing.
zerver/actions/user_groups.py
Outdated
@@ -278,9 +287,13 @@ def do_update_user_group_description( | |||
extra_data={ | |||
RealmAuditLog.OLD_VALUE: old_value, | |||
RealmAuditLog.NEW_VALUE: description, | |||
"property": "description", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed? We already have a different event type for description.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh.. right, removed!
@@ -2001,7 +2003,7 @@ def check_update_user_group( | |||
promote_new_full_members() | |||
check_update_user_group( | |||
"help", | |||
"Troubleshooting team", | |||
"Troubleshooting team\n", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to add a test for checking we don't embed url previews similar to the test we have for streams.
web/src/user_groups.ts
Outdated
group.description = event.data.description; | ||
group.rendered_description = event.data.rendered_description; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess clean_up_group_description
needs to be called here as well for the new value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no need to update the event.data.rendered_description
field here, doing group.rendered_description = clean_up_group_description(event.data.rendered_description)
will be better.
web/src/user_group_edit.ts
Outdated
rendered_description: postprocess_content(group.rendered_description), | ||
}); | ||
$edit_container.find(".group-description-wrapper").html(html); | ||
$group_row.find(".description").html(html); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to set the value rendered by render_user_group_description
here? This is for the left panel of groups UI and I guess that is handled by browse_user_groups_list_item
and it handles no description case using data-no-description
attribute.
render_typeahead_item({ | ||
primary: user_groups.get_display_group_name(user_group.name), | ||
secondary: user_group.description, | ||
secondary_html: user_group.rendered_description, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can see that the description is not aligned correctly with the name in mention typeahead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you please repopulate your dev db, this is because earlier created groups don't have clean_up_descriptions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There exists groups in production as well, so we cannot do that here. I missed this before but existing groups should be handled using migrations.
@@ -899,6 +902,13 @@ def create_system_user_groups_for_realm(realm: Realm) -> dict[str, NamedUserGrou | |||
everyone_on_internet_group_info, | |||
] | |||
|
|||
from zerver.actions.user_groups import render_group_description |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this seems fine for now then.
af67481
to
5437bb3
Compare
@sahil839 made the changes, PTAL. |
Hello, @sahil839! I've modified migration script, please take a look when you have time. |
from zerver.actions.user_groups import render_group_description | ||
|
||
NamedUserGroup = apps.get_model("zerver", "NamedUserGroup") | ||
all_named_user_groups = NamedUserGroup.objects.exclude(description="") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think batching might be needed here and probably updating the group objects in bulk as well like we do in migrations for setting defaults of group settings, for deploying this if there are large number of user groups having description.
Also, would be better to have this as a separate migration file for deployment ease.
web/src/user_groups.ts
Outdated
@@ -76,6 +80,13 @@ export function remove(user_group: UserGroup): void { | |||
user_group_by_id_dict.delete(user_group.id); | |||
} | |||
|
|||
export function clean_up_group_description(rendered_description: string): string { | |||
if (rendered_description !== undefined) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this be undefined because the type of rendered_description
above is string
.
web/src/user_groups.ts
Outdated
group.description = event.data.description; | ||
group.rendered_description = event.data.rendered_description; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no need to update the event.data.rendered_description
field here, doing group.rendered_description = clean_up_group_description(event.data.rendered_description)
will be better.
web/src/user_group_edit.ts
Outdated
@@ -469,7 +471,10 @@ export function handle_member_edit_event(group_id: number, user_ids: number[]): | |||
export function update_group_details(group: UserGroup): void { | |||
const $edit_container = get_edit_container(group); | |||
$edit_container.find(".group-name").text(user_groups.get_display_group_name(group.name)); | |||
$edit_container.find(".group-description").text(group.description); | |||
const html = render_user_group_description({ | |||
rendered_description: postprocess_content(group.rendered_description), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure why we do postprocess_content
only here when updating the description in right panel and not in other cases. I guess we do the same for streams as well, but we should check why is this required only here and not in other places.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks like render_browse_user_group_list_item
html somehow contains postprocessed content already, like with title,target etc. and postprocess_content
here doesn't really make any difference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Since we are passing the rendered template here, I guess this is not need since rendered_markdown
helper used in templates handles this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
found it!
there is this handlebars helper in template.ts
Handlebars.registerHelper(
"rendered_markdown",
(content: string) => new Handlebars.SafeString(postprocess_content(content)),
);
which already does postprocessing for rendered_desc everywhere (left_sidebar, right_sidebar, typeaheads, popovers), same for stream desc, so we don't really need this postprocess_content
.
also trying to figure out migration bit, but no luck yet, can't understand why the same thing worked for 0206_stream_rendered_description.py
:(
Hii @sahil839, I've made the changes, but getting an error while trying to migrate with new migration script. migration filefrom django.db import migrations, transaction
from django.db.backends.base.schema import BaseDatabaseSchemaEditor
from django.db.migrations.state import StateApps
from django.db.models import F, Max, Min, OuterRef
def render_all_user_group_descriptions(
apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
) -> None:
# FIXME: Application code should not be imported from migrations.
from zerver.actions.user_groups import render_group_description
NamedUserGroup = apps.get_model("zerver", "NamedUserGroup")
BATCH_SIZE = 1000
max_id = NamedUserGroup.objects.exclude(description="").aggregate(Max("id"))[
"id__max"
]
if max_id is None:
# Do nothing if there are no user groups on the server.
return
lower_bound = NamedUserGroup.objects.exclude(description="").aggregate(Min("id"))[
"id__min"
]
while lower_bound <= max_id:
upper_bound = lower_bound + BATCH_SIZE - 1
print(f"Processing batch {lower_bound} to {upper_bound} for NamedUserGroup")
with transaction.atomic():
all_named_user_groups = NamedUserGroup.objects.exclude(
id__gte=lower_bound, id__lt=upper_bound, description="")
for user_group in all_named_user_groups:
user_group.rendered_description = render_group_description(
user_group.description, user_group.realm
)
NamedUserGroup.objects.bulk_update(all_named_user_groups, fields=["rendered_description"], batch_size=BATCH_SIZE)
lower_bound += BATCH_SIZE
class Migration(migrations.Migration):
dependencies = [
("zerver", "0649_namedusergroup_rendered_description"),
]
operations = [
migrations.RunPython(
render_all_user_group_descriptions,
reverse_code=migrations.RunPython.noop,
elidable=True,
),
] Traceback
these groups seem to have no host so I added: realm_host = getattr(user_group.realm, 'host', None)
if not realm_host:
print(f"Skipping user group {user_group.name}: realm host is missing")
continue output is...Skipping user group role:nobody: realm host is missing Skipping user group role:owners: realm host is missing Skipping user group role:administrators: realm host is missing Skipping user group role:moderators: realm host is missing Skipping user group role:fullmembers: realm host is missing Skipping user group role:members: realm host is missing Skipping user group role:everyone: realm host is missing Skipping user group role:internet: realm host is missing Skipping user group role:nobody: realm host is missing Skipping user group role:owners: realm host is missing Skipping user group role:administrators: realm host is missing Skipping user group role:moderators: realm host is missing Skipping user group role:fullmembers: realm host is missing Skipping user group role:members: realm host is missing Skipping user group role:everyone: realm host is missing Skipping user group role:internet: realm host is missing Skipping user group hamletcharacters: realm host is missing Skipping user group role:nobody: realm host is missing Skipping user group role:owners: realm host is missing Skipping user group role:administrators: realm host is missing Skipping user group role:moderators: realm host is missing Skipping user group role:fullmembers: realm host is missing Skipping user group role:members: realm host is missing Skipping user group role:everyone: realm host is missing Skipping user group role:internet: realm host is missing Skipping user group new bolds: realm host is missing Skipping user group new italics: realm host is missing Skipping user group this is link: realm host is missingIf you have time can you please look into it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Posted a couple of comments.
Regarding migration, I tried running ./manage.py migrate
with the migration file that you have in this PR without any changes and it gives the same error that you mentioned. I think it is because uri
is a property on Realm
and not a DB field, but am not sure.
web/src/user_group_edit.ts
Outdated
@@ -469,7 +471,10 @@ export function handle_member_edit_event(group_id: number, user_ids: number[]): | |||
export function update_group_details(group: UserGroup): void { | |||
const $edit_container = get_edit_container(group); | |||
$edit_container.find(".group-name").text(user_groups.get_display_group_name(group.name)); | |||
$edit_container.find(".group-description").text(group.description); | |||
const html = render_user_group_description({ | |||
rendered_description: postprocess_content(group.rendered_description), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Since we are passing the rendered template here, I guess this is not need since rendered_markdown
helper used in templates handles this.
web/src/typeahead_helper.ts
Outdated
export let render_user_group = (user_group: { | ||
name: string; | ||
rendered_description: string; | ||
description: string; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
description
is not used anymore.
This commit adds support for rendered descriptions for user groups. changes: 1. Updated the NamedUserGroup model to include a rendered description. 2. Refactored backend functions to handle rendered descriptions for user groups. 3. Updated tests and documentation. 4. Added rendered descriptions to user group settings. 5. Group descriptions are saved and rendered on update. 6. Rendered descriptions are shown in places like: - Group list - Group Settings - Typeahead - Popovers The rendered description is also sent wherever the group's description is used. Fixes: zulip#32559
Thank you, @sahil839, for your time on this! I just figured out that running all migrations together ( |
You can do provisioning and rebuild the DB in development environment, but that cannot be done in production. Same migration file for stream description works, so am not sure how to fix this. |
Ok... then I'll continue to look for a fix, traceback
|
I meant I have not seen any previous discussion of any such failure when the stream rendered description work was done. The way you are trying to run the single migration run will expectedly not work because the Database structure has change a lot since that migration. I think would be better to ask for suggestions on CZO if you cannot get a fix. |
Heads up @reharsh, we just merged some commits that conflict with the changes you made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the |
This PR adds support for rendered descriptions for user groups.
Changes
The rendered description is also sent wherever the group's description is used.
Fixes: #32559
Screenshots
Screen Recordings
edit_recording.mov
Screen.Recording.2024-12-29.at.8.24.41.PM.mov
Self-review checklist
(variable names, code reuse, readability, etc.).
Communicate decisions, questions, and potential concerns.
Individual commits are ready for review (see commit discipline).
Completed manual review and testing of the following: