LogalDeveloper/owncast
0
Fork 0
Code Activity

Chat refactor + persistent backing chat users (#1163)

Browse Source 
* First pass at chat user registration and validation

* Disable chat if the user is disabled/blocked or the server hits max connections

* Handle dropping sockets if chat is disabled

* Fix origin in automated chat test

* Work for updated chat moderation

* Chat message markdown rendering and fix tests

* Put /api/chat behind a chat user access token. Closes #1085

* Reject blocked username changes

* More WIP moderation

* Defer configuring chat until we know if it is enabled. Closes #1135

* chat user blocking. Closes #1096

* Add tests around user access for #1096

* Add external integration chat message API + update integration auth middleware to pass along integration name. Closes #1092

* Delete old chat messages from db as to not hold on to excessive data. Closes #1152

* Add schema migration for messages. Closes #1155

* Commit updated API documentation

* Add chat load test

* Shared db mutex and db optimizations

* Simplify past display name handling

* Use a new test db for each test run

* Wire up the external messages actions + add tests for them

* Move access tokens to be actual users

* Run message pruning at launch + fix comparison

* Do not return API users in disabled users response

* Fix incorrect highlighting. Closes #1160

* Consolidate user table statements

* Set the max process connection limit to 70% of maximum

* Fix wrong old display name being returned in name change event

* Delete the old chat server files

* Wire back up the webhooks

* Remove unused

* Invalidate user cache on changes

* Do not send rendered body as RawBody

* Some cleanup

* Standardize names for external API users to ExternalAPIUser

* Do not log token

* Checkout branch when building admin for testing

* Bundle in dev admin for testing

* Some cleanup

* Cleanup js logs

* Cleanup and standardize event names

* Clean up some logging

* Update API spec. Closes #1133

* Commit updated API documentation

* Change paths to be better named

* Commit updated API documentation

* Update admin bundle

* Fix duplicate event name

* Rename scope var

* Update admin bundle

* Move connected clients controller into admin package

* Fix collecting usernames for autocomplete purposes

* No longer generate username when it is empty

* Sort clients and users by timestamp

* Move file to admin controller package

* Swap, so the comments stay correct

Co-authored-by: Jannik <jannik@outlook.com>

* Use explicit type alias

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented code.

Co-authored-by: Jannik <jannik@outlook.com>

* Cleanup test

* Remove some extra logging

* Add some clarity

* Update dev instance of admin for testing

* Consolidate lines

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented unused vars

Co-authored-by: Jannik <jannik@outlook.com>

* Until needed do not return IP address with client list

* Fix typo of wrong var

* Typo led to a bad test. Fix typo and fix test.

* Guard against the socket reconnecting on error if previously set to shutdown

* Do not log access tokens

* Return success message on enable/disable user

* Clean up some inactionable error messages. Sent ban message. Sort banned users.

* fix styling for when chat is completely disabled

* Unused

* guard against nil clients

* Update dev admin bundle

* Do not unhide messages when unblocking user just to be safe. Send removal action from the controller

* Add convinience function for getting active connections for a single user

* Lock db on these mutations

* Cleanup force disconnect using GetClientsForUser and capture client reference explicitly

* No longer re-showing banned user messages for safety. Removing this test.

* Remove no longer needed comment

* Tweaks to forbidden username handling.

- Standardize naming to not use "block" but "forbidden" instead.
- Pass array over the wire instead of string.
- Add API test
- Fix default list incorrectly being appended to custom list.

* Logging cleanup

* Update dev admin bundle

* Add an artificial delay in order to visually see message being hidden when testing

* Remove the user cache as it is a premature optimization

* When connected to chat let the user know their current user details to sync the username in the UI

* On connected send current display name back to client.
- Move name change out of chat component.
- Add additional event type constants.

* Fix broken workflow due to typo

* Troubleshoot workflow

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript (#1181)

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript

Bumps [htm](https://github.com/developit/htm) from 3.0.4 to 3.1.0.
- [Release notes](https://github.com/developit/htm/releases)
- [Commits](https://github.com/developit/htm/compare/3.0.4...3.1.0)

---
updated-dependencies:
- dependency-name: htm
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run npm run build and update libraries

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Gabe Kangas <gabek@real-ity.com>

* Commit updated Javascript packages

* Re-send current user info when a rejected name change takes place

* All socket writes should be through the send chan and not directly

* Seed the random generator

* Add keys and indexes to users table

* a util to generate consistent emoji markup

* console clean up

* mod tidy

* Commit updated API documentation

* Handle the max payload size of a socket message.
- Only close socket if x2 greater than the max size.
- Send the user a message if a message is too large.
- Surface the max size in bytes in the config.

* Update admin bundle

* Force all events to be sent in their own socket message and do not concatinate in a single message

* Update chat embed to register for access token

* Use a different access token for embed chat

* Update the chat message bubble background color to be bolder

* add base tag to open links in new window, closes #1220

* Support text input of :emoji: in chat (#1190)

* Initial implementation of emoji injection

* fix bookkeeping with multiple emoji

* make the emoji lookup case-insensitive

* try another solution for Caretposition

* add title to emojis

minor refactoring

* bind moji injection to InputKeyUp

* simplify the code

replace all found emojis

* inject emoji if the modifer is released earlier

* more efficient emoji tag search

* use json emoji.emoji as url

* use createEmojiMarkup()

* move emojify() to chat.js

* emojify on paste

* cleanup emoji titles in paste

* update inputText in InputKeyup

* mark emoji titles with 2*zwnj

this way paste cleanup will not interfere with text which include zwnj

* emoji should not change the inputText

* Do not show join messages when chat is offline. Closes #1224
- Show stream starting/ending messages in chat.
- When stream starts show everyone the welcome message.

* Force scrolling chat to bottom after history is populated regardless of scroll position. Closes https://github.com/owncast/owncast/issues/1222

* use maxSocketPayloadSize to calculate total bytes of message payload (#1221)

* utilize maxSocketPayloadSize from config; update chatInput to calculate based on that value instead of text value; remove usage of inputText for counting

* add a buffer to account for entire websocket payload for message char counting; trim nbsp;'s from ends of messages when calculating count

Co-authored-by: Gabe Kangas <gabek@real-ity.com>

Co-authored-by: Owncast <owncast@owncast.online>
Co-authored-by: Jannik <jannik@outlook.com>
Co-authored-by: Ginger Wong <omqmail@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Meisam <39205857+MFTabriz@users.noreply.github.com>
This commit is contained in:
Gabe Kangas
2021-07-19 19:22:29 -07:00
committed by GitHub
parent e3dc736cf4
commit b6f68628c0
88 changed files with 10691 additions and 2281 deletions
Download Patch File Download Diff File Expand all files Collapse all files

198
core/data/accessTokens.go
View File

@@ -1,198 +0,0 @@
package data
import (
"errors"
"strings"
"time"
"github.com/owncast/owncast/models"
log "github.com/sirupsen/logrus"
)
func createAccessTokensTable() {
log.Traceln("Creating access_tokens table...")
createTableSQL := `CREATE TABLE IF NOT EXISTS access_tokens (
"token" string NOT NULL PRIMARY KEY,
"name" string,
"scopes" TEXT,
"timestamp" DATETIME DEFAULT CURRENT_TIMESTAMP,
"last_used" DATETIME
);`
stmt, err := _db.Prepare(createTableSQL)
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
if _, err := stmt.Exec(); err != nil {
log.Warnln(err)
}
}
// InsertToken will add a new token to the database.
func InsertToken(token string, name string, scopes []string) error {
log.Println("Adding new access token:", name)
scopesString := strings.Join(scopes, ",")
tx, err := _db.Begin()
if err != nil {
return err
}
stmt, err := tx.Prepare("INSERT INTO access_tokens(token, name, scopes) values(?, ?, ?)")
if err != nil {
return err
}
defer stmt.Close()
if _, err := stmt.Exec(token, name, scopesString); err != nil {
return err
}
if err = tx.Commit(); err != nil {
return err
}
return nil
}
// DeleteToken will delete a token from the database.
func DeleteToken(token string) error {
log.Println("Deleting access token:", token)
tx, err := _db.Begin()
if err != nil {
return err
}
stmt, err := tx.Prepare("DELETE FROM access_tokens WHERE token = ?")
if err != nil {
return err
}
defer stmt.Close()
result, err := stmt.Exec(token)
if err != nil {
return err
}
if rowsDeleted, _ := result.RowsAffected(); rowsDeleted == 0 {
tx.Rollback() //nolint
return errors.New(token + " not found")
}
if err = tx.Commit(); err != nil {
return err
}
return nil
}
// DoesTokenSupportScope will determine if a specific token has access to perform a scoped action.
func DoesTokenSupportScope(token string, scope string) (bool, error) {
// This will split the scopes from comma separated to individual rows
// so we can efficiently find if a token supports a single scope.
// This is SQLite specific, so if we ever support other database
// backends we need to support other methods.
var query = `SELECT count(*) FROM (
WITH RECURSIVE split(token, scope, rest) AS (
SELECT token, '', scopes || ',' FROM access_tokens
UNION ALL
SELECT token,
substr(rest, 0, instr(rest, ',')),
substr(rest, instr(rest, ',')+1)
FROM split
WHERE rest <> '')
SELECT token, scope
FROM split
WHERE scope <> ''
ORDER BY token, scope
) AS token WHERE token.token = ? AND token.scope = ?;`
row := _db.QueryRow(query, token, scope)
var count = 0
err := row.Scan(&count)
return count > 0, err
}
// GetAccessTokens will return all access tokens.
func GetAccessTokens() ([]models.AccessToken, error) { //nolint
tokens := make([]models.AccessToken, 0)
// Get all messages sent within the past day
var query = "SELECT * FROM access_tokens"
rows, err := _db.Query(query)
if err != nil {
return tokens, err
}
defer rows.Close()
for rows.Next() {
var token string
var name string
var scopes string
var timestampString string
var lastUsedString *string
if err := rows.Scan(&token, &name, &scopes, &timestampString, &lastUsedString); err != nil {
log.Error("There is a problem reading the database.", err)
return tokens, err
}
timestamp, err := time.Parse(time.RFC3339, timestampString)
if err != nil {
return tokens, err
}
var lastUsed *time.Time = nil
if lastUsedString != nil {
lastUsedTime, _ := time.Parse(time.RFC3339, *lastUsedString)
lastUsed = &lastUsedTime
}
singleToken := models.AccessToken{
Name: name,
Token: token,
Scopes: strings.Split(scopes, ","),
Timestamp: timestamp,
LastUsed: lastUsed,
}
tokens = append(tokens, singleToken)
}
if err := rows.Err(); err != nil {
return tokens, err
}
return tokens, nil
}
// SetAccessTokenAsUsed will update the last used timestamp for a token.
func SetAccessTokenAsUsed(token string) error {
tx, err := _db.Begin()
if err != nil {
return err
}
stmt, err := tx.Prepare("UPDATE access_tokens SET last_used = CURRENT_TIMESTAMP WHERE token = ?")
if err != nil {
return err
}
defer stmt.Close()
if _, err := stmt.Exec(token); err != nil {
return err
}
if err = tx.Commit(); err != nil {
return err
}
return nil
}

26
core/data/config.go
View File

@@ -539,7 +539,9 @@ func VerifySettings() error {
if err := utils.Copy(defaultLogo, filepath.Join(config.DataDirectory, "logo.svg")); err != nil {
log.Errorln("error copying default logo: ", err)
}
SetLogoPath("logo.svg")
if err := SetLogoPath("logo.svg"); err != nil {
log.Errorln("unable to set default logo to logo.svg", err)
}
}
return nil
@@ -577,19 +579,25 @@ func FindHighestVideoQualityIndex(qualities []models.StreamOutputVariant) int {
return indexedQualities[0].index
}
// GetUsernameBlocklist will return the blocked usernames as a comma separated string.
func GetUsernameBlocklist() string {
// GetForbiddenUsernameList will return the blocked usernames as a comma separated string.
func GetForbiddenUsernameList() []string {
usernameString, err := _datastore.GetString(blockedUsernamesKey)
if err != nil {
log.Traceln(blockedUsernamesKey, err)
return ""
return config.DefaultForbiddenUsernames
}
return usernameString
if usernameString == "" {
return config.DefaultForbiddenUsernames
}
blocklist := strings.Split(usernameString, ",")
return blocklist
}
// SetUsernameBlocklist set the username blocklist as a comma separated string.
func SetUsernameBlocklist(usernames string) error {
return _datastore.SetString(blockedUsernamesKey, usernames)
// SetForbiddenUsernameList set the username blocklist as a comma separated string.
func SetForbiddenUsernameList(usernames []string) error {
usernameListString := strings.Join(usernames, ",")
return _datastore.SetString(blockedUsernamesKey, usernameListString)
}

20
core/data/data.go
View File

@@ -17,7 +17,7 @@ import (
)
const (
schemaVersion = 0
schemaVersion = 1
)
var _db *sql.DB
@@ -45,7 +45,13 @@ func SetupPersistence(file string) error {
}
}
db, err := sql.Open("sqlite3", file)
db, err := sql.Open("sqlite3", fmt.Sprintf("file:%s", file))
db.SetMaxOpenConns(1)
_db = db
createWebhooksTable()
createUsersTable(db)
if err != nil {
return err
}
@@ -86,11 +92,6 @@ func SetupPersistence(file string) error {
}
}
_db = db
createWebhooksTable()
createAccessTokensTable()
_datastore = &Datastore{}
_datastore.Setup()
@@ -106,13 +107,14 @@ func SetupPersistence(file string) error {
}
func migrateDatabase(db *sql.DB, from, to int) error {
log.Printf("Migrating database from version %d to %d\n", from, to)
log.Printf("Migrating database from version %d to %d", from, to)
dbBackupFile := filepath.Join(config.BackupDirectory, fmt.Sprintf("owncast-v%d.bak", from))
utils.Backup(db, dbBackupFile)
for v := from; v < to; v++ {
switch v {
case 0:
log.Printf("Migration step from %d to %d\n", v, v+1)
log.Printf("Migration step from %d to %d", v, v+1)
migrateToSchema1(db)
default:
panic("missing database migration step")
}

9
core/data/data_test.go
View File

@@ -2,13 +2,18 @@ package data
import (
"fmt"
"io/ioutil"
"os"
"testing"
)
func TestMain(m *testing.M) {
dbFile := "../../test/test.db"
dbFile, err := ioutil.TempFile(os.TempDir(), "owncast-test-db.db")
if err != nil {
panic(err)
}
SetupPersistence(dbFile)
SetupPersistence(dbFile.Name())
m.Run()
}

118
core/data/migrations.go Normal file
View File

@@ -0,0 +1,118 @@
package data
import (
"database/sql"
"time"
"github.com/owncast/owncast/utils"
log "github.com/sirupsen/logrus"
"github.com/teris-io/shortid"
)
func migrateToSchema1(db *sql.DB) {
// Since it's just a backlog of chat messages let's wipe the old messages
// and recreate the table.
// Drop the old messages table
stmt, err := db.Prepare("DROP TABLE messages")
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
_, err = stmt.Exec()
if err != nil {
log.Warnln(err)
}
// Recreate it
createUsersTable(db)
// Migrate access tokens to become chat users
type oldAccessToken struct {
accessToken string
displayName string
scopes string
createdAt time.Time
lastUsedAt *time.Time
}
oldAccessTokens := make([]oldAccessToken, 0)
query := `SELECT * FROM access_tokens`
rows, err := db.Query(query)
if err != nil || rows.Err() != nil {
log.Errorln("error migrating access tokens to schema v1", err, rows.Err())
return
}
defer rows.Close()
for rows.Next() {
var token string
var name string
var scopes string
var timestampString string
var lastUsedString *string
if err := rows.Scan(&token, &name, &scopes, &timestampString, &lastUsedString); err != nil {
log.Error("There is a problem reading the database.", err)
return
}
timestamp, err := time.Parse(time.RFC3339, timestampString)
if err != nil {
return
}
var lastUsed *time.Time = nil
if lastUsedString != nil {
lastUsedTime, _ := time.Parse(time.RFC3339, *lastUsedString)
lastUsed = &lastUsedTime
}
oldToken := oldAccessToken{
accessToken: token,
displayName: name,
scopes: scopes,
createdAt: timestamp,
lastUsedAt: lastUsed,
}
oldAccessTokens = append(oldAccessTokens, oldToken)
}
// Recreate them as users
for _, token := range oldAccessTokens {
color := utils.GenerateRandomDisplayColor()
if err := insertAPIToken(db, token.accessToken, token.displayName, color, token.scopes); err != nil {
log.Errorln("Error migrating access token", err)
}
}
}
func insertAPIToken(db *sql.DB, token string, name string, color int, scopes string) error {
log.Debugln("Adding new access token:", name)
id := shortid.MustGenerate()
tx, err := db.Begin()
if err != nil {
return err
}
stmt, err := tx.Prepare("INSERT INTO users(id, access_token, display_name, display_color, scopes, type) values(?, ?, ?, ?, ?, ?)")
if err != nil {
return err
}
defer stmt.Close()
if _, err = stmt.Exec(id, token, name, color, scopes, "API"); err != nil {
return err
}
if err = tx.Commit(); err != nil {
return err
}
return nil
}

45
core/data/persistence.go
View File

@@ -4,6 +4,7 @@ import (
"bytes"
"database/sql"
"encoding/gob"
"sync"
// sqlite requires a blank import.
_ "github.com/mattn/go-sqlite3"
@@ -12,14 +13,15 @@ import (
// Datastore is the global key/value store for configuration values.
type Datastore struct {
db *sql.DB
cache map[string][]byte
DB *sql.DB
cache map[string][]byte
DbLock *sync.Mutex
}
func (ds *Datastore) warmCache() {
log.Traceln("Warming config value cache")
res, err := ds.db.Query("SELECT key, value FROM datastore")
res, err := ds.DB.Query("SELECT key, value FROM datastore")
if err != nil || res.Err() != nil {
log.Errorln("error warming config cache", err, res.Err())
}
@@ -48,7 +50,7 @@ func (ds *Datastore) Get(key string) (ConfigEntry, error) {
var resultKey string
var resultValue []byte
row := ds.db.QueryRow("SELECT key, value FROM datastore WHERE key = ? LIMIT 1", key)
row := ds.DB.QueryRow("SELECT key, value FROM datastore WHERE key = ? LIMIT 1", key)
if err := row.Scan(&resultKey, &resultValue); err != nil {
return ConfigEntry{}, err
}
@@ -63,36 +65,26 @@ func (ds *Datastore) Get(key string) (ConfigEntry, error) {
// Save will save the ConfigEntry to the database.
func (ds *Datastore) Save(e ConfigEntry) error {
ds.DbLock.Lock()
defer ds.DbLock.Unlock()
var dataGob bytes.Buffer
enc := gob.NewEncoder(&dataGob)
if err := enc.Encode(e.Value); err != nil {
return err
}
tx, err := ds.db.Begin()
tx, err := ds.DB.Begin()
if err != nil {
return err
}
var stmt *sql.Stmt
var count int
row := ds.db.QueryRow("SELECT COUNT(*) FROM datastore WHERE key = ? LIMIT 1", e.Key)
if err := row.Scan(&count); err != nil {
stmt, err = tx.Prepare("INSERT INTO datastore (key, value) VALUES(?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value")
if err != nil {
return err
}
_, err = stmt.Exec(e.Key, dataGob.Bytes())
if count == 0 {
stmt, err = tx.Prepare("INSERT INTO datastore(key, value) values(?, ?)")
if err != nil {
return err
}
_, err = stmt.Exec(e.Key, dataGob.Bytes())
} else {
stmt, err = tx.Prepare("UPDATE datastore SET value=? WHERE key=?")
if err != nil {
return err
}
_, err = stmt.Exec(dataGob.Bytes(), e.Key)
}
if err != nil {
return err
}
@@ -110,7 +102,8 @@ func (ds *Datastore) Save(e ConfigEntry) error {
// Setup will create the datastore table and perform initial initialization.
func (ds *Datastore) Setup() {
ds.cache = make(map[string][]byte)
ds.db = GetDatabase()
ds.DB = GetDatabase()
ds.DbLock = &sync.Mutex{}
createTableSQL := `CREATE TABLE IF NOT EXISTS datastore (
"key" string NOT NULL PRIMARY KEY,
@@ -118,7 +111,7 @@ func (ds *Datastore) Setup() {
"timestamp" DATE DEFAULT CURRENT_TIMESTAMP NOT NULL
);`
stmt, err := ds.db.Prepare(createTableSQL)
stmt, err := ds.DB.Prepare(createTableSQL)
if err != nil {
log.Fatal(err)
}
@@ -137,7 +130,7 @@ func (ds *Datastore) Setup() {
// Reset will delete all config entries in the datastore and start over.
func (ds *Datastore) Reset() {
sql := "DELETE FROM datastore"
stmt, err := ds.db.Prepare(sql)
stmt, err := ds.DB.Prepare(sql)
if err != nil {
log.Fatalln(err)
}
@@ -150,3 +143,7 @@ func (ds *Datastore) Reset() {
PopulateDefaults()
}
func GetDatastore() *Datastore {
return _datastore
}

37
core/data/users.go Normal file
View File

@@ -0,0 +1,37 @@
package data
import (
"database/sql"
log "github.com/sirupsen/logrus"
)
func createUsersTable(db *sql.DB) {
log.Traceln("Creating users table...")
createTableSQL := `CREATE TABLE IF NOT EXISTS users (
"id" TEXT,
"access_token" string NOT NULL,
"display_name" TEXT NOT NULL,
"display_color" NUMBER NOT NULL,
"created_at" TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
"disabled_at" TIMESTAMP,
"previous_names" TEXT DEFAULT '',
"namechanged_at" TIMESTAMP,
"scopes" TEXT,
"type" TEXT DEFAULT 'STANDARD',
"last_used" DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id, access_token),
UNIQUE(id, access_token)
);CREATE INDEX index ON users (id, access_token)`
stmt, err := db.Prepare(createTableSQL)
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
_, err = stmt.Exec()
if err != nil {
log.Warnln(err)
}
}

6
core/data/webhooks.go
View File

@@ -33,7 +33,7 @@ func createWebhooksTable() {
// InsertWebhook will add a new webhook to the database.
func InsertWebhook(url string, events []models.EventType) (int, error) {
log.Println("Adding new webhook:", url)
log.Traceln("Adding new webhook:", url)
eventsString := strings.Join(events, ",")
@@ -67,7 +67,7 @@ func InsertWebhook(url string, events []models.EventType) (int, error) {
// DeleteWebhook will delete a webhook from the database.
func DeleteWebhook(id int) error {
log.Println("Deleting webhook:", id)
log.Traceln("Deleting webhook:", id)
tx, err := _db.Begin()
if err != nil {
@@ -86,7 +86,7 @@ func DeleteWebhook(id int) error {
}
if rowsDeleted, _ := result.RowsAffected(); rowsDeleted == 0 {
tx.Rollback() //nolint
_ = tx.Rollback()
return errors.New(fmt.Sprint(id) + " not found")
}