Chat refactor + persistent backing chat users (#1163)

* First pass at chat user registration and validation

* Disable chat if the user is disabled/blocked or the server hits max connections

* Handle dropping sockets if chat is disabled

* Fix origin in automated chat test

* Work for updated chat moderation

* Chat message markdown rendering and fix tests

* Put /api/chat behind a chat user access token. Closes #1085

* Reject blocked username changes

* More WIP moderation

* Defer configuring chat until we know if it is enabled. Closes #1135

* chat user blocking. Closes #1096

* Add tests around user access for #1096

* Add external integration chat message API + update integration auth middleware to pass along integration name. Closes #1092

* Delete old chat messages from db as to not hold on to excessive data. Closes #1152

* Add schema migration for messages. Closes #1155

* Commit updated API documentation

* Add chat load test

* Shared db mutex and db optimizations

* Simplify past display name handling

* Use a new test db for each test run

* Wire up the external messages actions + add tests for them

* Move access tokens to be actual users

* Run message pruning at launch + fix comparison

* Do not return API users in disabled users response

* Fix incorrect highlighting. Closes #1160

* Consolidate user table statements

* Set the max process connection limit to 70% of maximum

* Fix wrong old display name being returned in name change event

* Delete the old chat server files

* Wire back up the webhooks

* Remove unused

* Invalidate user cache on changes

* Do not send rendered body as RawBody

* Some cleanup

* Standardize names for external API users to ExternalAPIUser

* Do not log token

* Checkout branch when building admin for testing

* Bundle in dev admin for testing

* Some cleanup

* Cleanup js logs

* Cleanup and standardize event names

* Clean up some logging

* Update API spec. Closes #1133

* Commit updated API documentation

* Change paths to be better named

* Commit updated API documentation

* Update admin bundle

* Fix duplicate event name

* Rename scope var

* Update admin bundle

* Move connected clients controller into admin package

* Fix collecting usernames for autocomplete purposes

* No longer generate username when it is empty

* Sort clients and users by timestamp

* Move file to admin controller package

* Swap, so the comments stay correct

Co-authored-by: Jannik <jannik@outlook.com>

* Use explicit type alias

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented code.

Co-authored-by: Jannik <jannik@outlook.com>

* Cleanup test

* Remove some extra logging

* Add some clarity

* Update dev instance of admin for testing

* Consolidate lines

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented unused vars

Co-authored-by: Jannik <jannik@outlook.com>

* Until needed do not return IP address with client list

* Fix typo of wrong var

* Typo led to a bad test. Fix typo and fix test.

* Guard against the socket reconnecting on error if previously set to shutdown

* Do not log access tokens

* Return success message on enable/disable user

* Clean up some inactionable error messages. Sent ban message. Sort banned users.

* fix styling for when chat is completely disabled

* Unused

* guard against nil clients

* Update dev admin bundle

* Do not unhide messages when unblocking user just to be safe. Send removal action from the controller

* Add convinience function for getting active connections for a single user

* Lock db on these mutations

* Cleanup force disconnect using GetClientsForUser and capture client reference explicitly

* No longer re-showing banned user messages for safety. Removing this test.

* Remove no longer needed comment

* Tweaks to forbidden username handling.

- Standardize naming to not use "block" but "forbidden" instead.
- Pass array over the wire instead of string.
- Add API test
- Fix default list incorrectly being appended to custom list.

* Logging cleanup

* Update dev admin bundle

* Add an artificial delay in order to visually see message being hidden when testing

* Remove the user cache as it is a premature optimization

* When connected to chat let the user know their current user details to sync the username in the UI

* On connected send current display name back to client.
- Move name change out of chat component.
- Add additional event type constants.

* Fix broken workflow due to typo

* Troubleshoot workflow

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript (#1181)

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript

Bumps [htm](https://github.com/developit/htm) from 3.0.4 to 3.1.0.
- [Release notes](https://github.com/developit/htm/releases)
- [Commits](https://github.com/developit/htm/compare/3.0.4...3.1.0)

---
updated-dependencies:
- dependency-name: htm
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run npm run build and update libraries

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Gabe Kangas <gabek@real-ity.com>

* Commit updated Javascript packages

* Re-send current user info when a rejected name change takes place

* All socket writes should be through the send chan and not directly

* Seed the random generator

* Add keys and indexes to users table

* a util to generate consistent emoji markup

* console clean up

* mod tidy

* Commit updated API documentation

* Handle the max payload size of a socket message.
- Only close socket if x2 greater than the max size.
- Send the user a message if a message is too large.
- Surface the max size in bytes in the config.

* Update admin bundle

* Force all events to be sent in their own socket message and do not concatinate in a single message

* Update chat embed to register for access token

* Use a different access token for embed chat

* Update the chat message bubble background color to be bolder

* add base tag to open links in new window, closes #1220

* Support text input of :emoji: in chat (#1190)

* Initial implementation of emoji injection

* fix bookkeeping with multiple emoji

* make the emoji lookup case-insensitive

* try another solution for Caretposition

* add title to emojis

minor refactoring

* bind moji injection to InputKeyUp

* simplify the code

replace all found emojis

* inject emoji if the modifer is released earlier

* more efficient emoji tag search

* use json emoji.emoji as url

* use createEmojiMarkup()

* move emojify() to chat.js

* emojify on paste

* cleanup emoji titles in paste

* update inputText in InputKeyup

* mark emoji titles with 2*zwnj

this way paste cleanup will not interfere with text which include zwnj

* emoji should not change the inputText

* Do not show join messages when chat is offline. Closes #1224
- Show stream starting/ending messages in chat.
- When stream starts show everyone the welcome message.

* Force scrolling chat to bottom after history is populated regardless of scroll position. Closes https://github.com/owncast/owncast/issues/1222

* use maxSocketPayloadSize to calculate total bytes of message payload (#1221)

* utilize maxSocketPayloadSize from config; update chatInput to calculate based on that value instead of text value; remove usage of inputText for counting

* add a buffer to account for entire websocket payload for message char counting; trim nbsp;'s from ends of messages when calculating count

Co-authored-by: Gabe Kangas <gabek@real-ity.com>

Co-authored-by: Owncast <owncast@owncast.online>
Co-authored-by: Jannik <jannik@outlook.com>
Co-authored-by: Ginger Wong <omqmail@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Meisam <39205857+MFTabriz@users.noreply.github.com>

core/chat/events/actionEvent.go

@@ -0,0 +1,20 @@
+package events
+
+type ActionEvent struct {
+	Event
+	MessageEvent
+}
+
+// ActionEvent will return the object to send to all chat users.
+func (e *ActionEvent) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"body":      e.Body,
+		"type":      e.GetMessageType(),
+	}
+}
+
+func (e *ActionEvent) GetMessageType() EventType {
+	return ChatActionSent
+}

core/chat/events/events.go

@@ -0,0 +1,156 @@
+package events
+
+import (
+	"bytes"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/microcosm-cc/bluemonday"
+	"github.com/teris-io/shortid"
+	"github.com/yuin/goldmark"
+	"github.com/yuin/goldmark/extension"
+	"github.com/yuin/goldmark/renderer/html"
+	"mvdan.cc/xurls"
+
+	"github.com/owncast/owncast/core/user"
+	log "github.com/sirupsen/logrus"
+)
+
+// EventPayload is a generic key/value map for sending out to chat clients.
+type EventPayload map[string]interface{}
+
+type OutboundEvent interface {
+	GetBroadcastPayload() EventPayload
+	GetMessageType() EventType
+}
+
+// Event is any kind of event.  A type is required to be specified.
+type Event struct {
+	Type      EventType `json:"type"`
+	Id        string    `json:"id"`
+	Timestamp time.Time `json:"timestamp"`
+}
+
+type UserEvent struct {
+	User     *user.User `json:"user"`
+	HiddenAt *time.Time `json:"hiddenAt,omitempty"`
+}
+
+// MessageEvent is an event that has a message body.
+type MessageEvent struct {
+	OutboundEvent `json:"-"`
+	Body          string `json:"body"`
+	RawBody       string `json:"-"`
+}
+
+type SystemActionEvent struct {
+	Event
+	MessageEvent
+}
+
+// SetDefaults will set default properties of all inbound events.
+func (e *Event) SetDefaults() {
+	e.Id = shortid.MustGenerate()
+	e.Timestamp = time.Now()
+}
+
+// SetDefaults will set default properties of all inbound events.
+func (e *UserMessageEvent) SetDefaults() {
+	e.Id = shortid.MustGenerate()
+	e.Timestamp = time.Now()
+	e.RenderAndSanitizeMessageBody()
+}
+
+// RenderAndSanitizeMessageBody will turn markdown into HTML, sanitize raw user-supplied HTML and standardize
+// the message into something safe and renderable for clients.
+func (m *MessageEvent) RenderAndSanitizeMessageBody() {
+	m.RawBody = m.Body
+
+	// Set the new, sanitized and rendered message body
+	m.Body = RenderAndSanitize(m.RawBody)
+}
+
+// Empty will return if this message's contents is empty.
+func (m *MessageEvent) Empty() bool {
+	return m.Body == ""
+}
+
+// RenderBody will render markdown to html without any sanitization.
+func (m *MessageEvent) RenderBody() {
+	m.RawBody = m.Body
+	m.Body = RenderMarkdown(m.RawBody)
+}
+
+// RenderAndSanitize will turn markdown into HTML, sanitize raw user-supplied HTML and standardize
+// the message into something safe and renderable for clients.
+func RenderAndSanitize(raw string) string {
+	rendered := RenderMarkdown(raw)
+	safe := sanitize(rendered)
+
+	// Set the new, sanitized and rendered message body
+	return strings.TrimSpace(safe)
+}
+
+func RenderMarkdown(raw string) string {
+	markdown := goldmark.New(
+		goldmark.WithRendererOptions(
+			html.WithUnsafe(),
+		),
+		goldmark.WithExtensions(
+			extension.NewLinkify(
+				extension.WithLinkifyAllowedProtocols([][]byte{
+					[]byte("http:"),
+					[]byte("https:"),
+				}),
+				extension.WithLinkifyURLRegexp(
+					xurls.Strict,
+				),
+			),
+		),
+	)
+
+	trimmed := strings.TrimSpace(raw)
+	var buf bytes.Buffer
+	if err := markdown.Convert([]byte(trimmed), &buf); err != nil {
+		log.Debugln(err)
+	}
+
+	return buf.String()
+}
+
+func sanitize(raw string) string {
+	p := bluemonday.StrictPolicy()
+
+	// Require URLs to be parseable by net/url.Parse
+	p.AllowStandardURLs()
+	p.RequireParseableURLs(true)
+
+	// Allow links
+	p.AllowAttrs("href").OnElements("a")
+
+	// Force all URLs to have "noreferrer" in their rel attribute.
+	p.RequireNoReferrerOnLinks(true)
+
+	// Links will get target="_blank" added to them.
+	p.AddTargetBlankToFullyQualifiedLinks(true)
+
+	// Allow breaks
+	p.AllowElements("br", "p")
+
+	// Allow img tags from the the local emoji directory only
+	p.AllowAttrs("src", "alt", "class", "title").Matching(regexp.MustCompile(`(?i)/img/emoji`)).OnElements("img")
+	p.AllowAttrs("class").OnElements("img")
+
+	// Allow bold
+	p.AllowElements("strong")
+
+	// Allow emphasis
+	p.AllowElements("em")
+
+	// Allow code blocks
+	p.AllowElements("code")
+	p.AllowElements("pre")
+
+	return p.Sanitize(raw)
+}

core/chat/events/eventtype.go

@@ -0,0 +1,34 @@
+package events
+
+// EventType is the type of a websocket event.
+type EventType = string
+
+const (
+	// MessageSent is the event sent when a chat event takes place.
+	MessageSent EventType = "CHAT"
+	// UserJoined is the event sent when a chat user join action takes place.
+	UserJoined EventType = "USER_JOINED"
+	// UserNameChanged is the event sent when a chat username change takes place.
+	UserNameChanged EventType = "NAME_CHANGE"
+	// VisibiltyToggled is the event sent when a chat message's visibility changes.
+	VisibiltyToggled EventType = "VISIBILITY-UPDATE"
+	// PING is a ping message.
+	PING EventType = "PING"
+	// PONG is a pong message.
+	PONG EventType = "PONG"
+	// StreamStarted represents a stream started event.
+	StreamStarted EventType = "STREAM_STARTED"
+	// StreamStopped represents a stream stopped event.
+	StreamStopped EventType = "STREAM_STOPPED"
+	// SystemMessageSent is the event sent when a system message is sent.
+	SystemMessageSent EventType = "SYSTEM"
+	// ChatDisabled is when a user is explicitly disabled and blocked from using chat.
+	ChatDisabled EventType = "CHAT_DISABLED"
+	// ConnectedUserInfo is a private event to a user letting them know their user details.
+	ConnectedUserInfo EventType = "CONNECTED_USER_INFO"
+	// ChatActionSent is a generic chat action that can be used for anything that doesn't need specific handling or formatting.
+	ChatActionSent              EventType = "CHAT_ACTION"
+	ErrorNeedsRegistration      EventType = "ERROR_NEEDS_REGISTRATION"
+	ErrorMaxConnectionsExceeded EventType = "ERROR_MAX_CONNECTIONS_EXCEEDED"
+	ErrorUserDisabled           EventType = "ERROR_USER_DISABLED"
+)

core/chat/events/nameChangeEvent.go

@@ -0,0 +1,26 @@
+package events
+
+// NameChangeEvent is received when a user changes their chat display name.
+type NameChangeEvent struct {
+	Event
+	UserEvent
+	NewName string `json:"newName"`
+}
+
+// NameChangeEventBroadcast is fired when a user changes their chat display name.
+type NameChangeBroadcast struct {
+	Event
+	UserEvent
+	Oldname string `json:"oldName"`
+}
+
+// GetBroadcastPayload will return the object to send to all chat users.
+func (e *NameChangeBroadcast) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"user":      e.User,
+		"oldName":   e.Oldname,
+		"type":      UserNameChanged,
+	}
+}

core/chat/events/systemMessageEvent.go

@@ -0,0 +1,26 @@
+package events
+
+import "github.com/owncast/owncast/core/data"
+
+// SystemMessageEvent is a message displayed in chat on behalf of the server.
+type SystemMessageEvent struct {
+	Event
+	MessageEvent
+}
+
+// SystemMessageEvent will return the object to send to all chat users.
+func (e *SystemMessageEvent) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"body":      e.Body,
+		"type":      SystemMessageSent,
+		"user": EventPayload{
+			"displayName": data.GetServerName(),
+		},
+	}
+}
+
+func (e *SystemMessageEvent) GetMessageType() EventType {
+	return SystemMessageSent
+}

core/chat/events/userDisabledEvent.go

@@ -0,0 +1,17 @@
+package events
+
+// UserDisabledEvent is the event fired when a user is banned/blocked and disconnected from chat.
+type UserDisabledEvent struct {
+	Event
+	UserEvent
+}
+
+// GetBroadcastPayload will return the object to send to all chat users.
+func (e *UserDisabledEvent) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"type":      ErrorUserDisabled,
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"user":      e.User,
+	}
+}

core/chat/events/userJoinedEvent.go

@@ -0,0 +1,17 @@
+package events
+
+// UserJoinedEvent is the event fired when a user joins chat.
+type UserJoinedEvent struct {
+	Event
+	UserEvent
+}
+
+// GetBroadcastPayload will return the object to send to all chat users.
+func (e *UserJoinedEvent) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"type":      UserJoined,
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"user":      e.User,
+	}
+}

core/chat/events/userMessageEvent.go

@@ -0,0 +1,24 @@
+package events
+
+// UserMessageEvent is an inbound message from a user.
+type UserMessageEvent struct {
+	Event
+	UserEvent
+	MessageEvent
+}
+
+// GetBroadcastPayload will return the object to send to all chat users.
+func (e *UserMessageEvent) GetBroadcastPayload() EventPayload {
+	return EventPayload{
+		"id":        e.Id,
+		"timestamp": e.Timestamp,
+		"body":      e.Body,
+		"user":      e.User,
+		"type":      MessageSent,
+		"visible":   e.HiddenAt == nil,
+	}
+}
+
+func (e *UserMessageEvent) GetMessageType() EventType {
+	return MessageSent
+}