Social features / ActivityPub federation (#1629)

* Support webfinger requests for the live account. Closes https://github.com/owncast/owncast/issues/1193

* Support for actor requests. Returns response for live actor. Closes https://github.com/owncast/owncast/issues/1203

* Handle follow and unfollow requests. Closes
https://github.com/owncast/owncast/issues/1191 and https://github.com/owncast/owncast/issues/1205 and https://github.com/owncast/owncast/issues/1206 and https://github.com/owncast/owncast/issues/1194

* Add basic support for sending out text activities. For https://github.com/owncast/owncast/issues/1192

* Some error handling and passing of dynamic local account names.

* Add hardcoded example image attachment to test post

* Centralize the map of accounts and inboxes

* No longer disable the preview generator based on YP toggle

* Send a federated message to followers when stream starts. For https://github.com/owncast/owncast/issues/1192

* Placeholder for attaching tags

* Add image description

* Save and get to outbox persistence. Return using outbox endpoint for actor

* Pass payloads to be handled through the gochan

* Handle undo follow requests explitly, not all undo requests

* Add API for manually sending simple federated messages. Closes #1215

* Verify inbox requests. Closes #1321

* Add route to fetch a single AP object by ID. For #1329

* Add responses to fediverse nodeinfo requests

* Set and get federation config values for admin

* Handle host-meta requests

* Do not send out message if disabled. Use saved go live message.

* Require AP-compatible content types for AP-related requests

* Rename ap models to apmodels for clarity

* Change how content type matching takes place.

* io -> ioutil

* Add stub delete activity callback

* Handle likes and announces to surface engagement in chat. Part of #1229

* Append url to go live posts

* Do not require specific content types for nodeinfo requests

* Add follow engagement chat message via AP

* add owncast user-agent to requests

* Set note visibility to public (for now)

* Fix saving/fetching a single object

* Add support for x-nodeinfo2 responses

* Point to the dev admin branch for ap

* Bundle in dev admin for testing

* Add error logging

* Add AP middleware back

* Point to the new external compatible logo endpoint

* Clean up more AP logging to help testing

* Tweak go live text and link hashtags

* Fix bug in fetching init time

* Send update actor activities when server details/profile is updated

* Add federation config overview to web client config

* Add additional actor properties

* Make the AP middleware checking more flexible when looking at types

* First pass at remote fediverse follow flow. For #1371

* Added a basic AP actor followers endpoint

* WIP client followers API

* Add profile-page reference to webfinger response

* Add aliases to webfinger response

* Fix content-type returned to be expected activitypub+json

* First pass at followers api

* Point at local dev copy of go-fed/activity

* Add custom toot Hashtag objects to posts

* Store additional user details to followers table

* Fix AP followers endpoint. Closes #1204

* Add owncast hashtag as an invisible tag to go live posts

* Reject AP requests when it is disabled

* Add actor util for generating full account user from person object

* Verify inbox requests before performing any other work

* Accept actor update requests

* Fix linter errors in federation branch

* Migrate AP SQL to sqlc for type safe queries

* Use the @unclearParadigm REST parameter helper

* Fix verifying post ID on AP engagement

* WIP privacy/request approval

* Style the remote follow modal

* First pass at a followers list component w/ mock data. #1370

* Revert "Use the @unclearParadigm REST parameter helper"

This reverts commit c8af8a413f6f53e7d1a15a7d823ff28be2db3c23.

* Fix get followers API

* Add support for requiring approval. Closes https://github.com/owncast/owncast/issues/1208

* Handle Applications as Actors partly for PeerTube support

* add temp todo list

* check route on load, this might change later

* style followers

* account for just 1 tab case

* Remove mock data. Allow showing follow button even when there are no external actions defined

* Point to actual followers API

* Support fallback img for follower views

* Remove duplicate verification. Add some additional verbose logging

* Bundle dev admin

* Add type to host-meta webfinger template response

* Tweak remote follow modal content

* WIP federation followers refactor

* Do not send pointer to middleware

* Update admin

* Add setting for toggling displaying fediverse engagement. Closes #1404

* Add in-development admin

* Do not enable cors on admin followers api

* Add db migration for updating messages table

* Enable empty string go live messages to disable

* Remove debug messages

* Rework some ActivityPub handling.

Create new Actor->Person handling.
Create new Actor->Service handling.
Add engagement handlers to send chat events and store event objects.
Store inbound activities to new ap_inbound_activities table.

* Support federated engagement events.

Store them in the messages table and surface them via chat events.

* Support federated event engatement in the chat

* Tweak web UI followers handling

* Point go.mod at remote fork instead of local

* Update admin

* Merged in develop. Couple fixes

* Update dev admin

* Update fedi engagement posts.

- Fix incorrect action text.
- Add action icons.

* Set public as to instead of cc for ap msg

* Updated styling for federated actions in chat

* Add support for blocking federated domains. Closes #1209

* Force checking of https in verify step

* Update dev admin

* Return user scopes in chat history api. Closes #1586

* Update dev admin

* Add AP outbound request worker pool. Closes #1571

* Disable (temporarily?) owncast tag on AP posts

* Consolidate creating activity+notes in outbound AP messages

* Add inbox worker pool. Closes #1570

* Update dev admin bundle

* Clean up some logs

* Re-enable inbound verfication

* Save full IRI to outbox instead of path

* Reject if full IRI is not found in outbox

* Use full ActivityPub user account in chat event

* Fix and expand follower APIs

- Add missing IDs to AP follower endpoints
- Split AP follower endpoints into initial request and pages.
- Support pagination in AP requests.

* Include IRI in error message

* Hide chat toggle when chat is hidden. Closes #1606

* Updates to followers pagination

* Set default go live message

* Remove log

* indirect -> direct import

* Updates for inbound federated event handling.

- Keep track of existing events and reject duplicates.
- Change what is sent to chat for surfing federated engagement.
- Keep track if outbound events are automated "go live" events or not.

* Update chat federated engagement.

* Update dev admin.

* Move from being a person to a bot (service). Closes #1619

* Only set server init date if not already set

* Only save notes to outbox able

* Rework private-mode followers/approvals

* API for returning a list of federated actions for #1573

* Fix too-small follower cells and jumpy tabs. Closes #1616 and closes #1516

* Fix shortcuts getting fired on inputs. Fixes #1489 and #1201

* Add spinner, autoclose + other fixes to follow modal. Fixes #1593

* Fix fetching a single object by IRI

* SendFederationMessage -> SendFederatedMessage

* Autolink and create tag objects from manual posts. Closes #1620

* Update dev admin bundle

* Handle engagement from non-automated/live posts

* Reject federated engagement actions if they do not match a local post

* Update dev admin bundle

* A bunch of cleanup

* Fix unused assignments and logic

* Remove unused function

* Add content warning and sentive content flag if stream is NSFW. Closes #1624

* Disable fetching objects by IRI when in private mode. Closes #1623

* Update the error message of the remote follow dialog. closes #1622

* Update dev admin

* Fix NREs throwing in test content

* Fix query that wasn't properly filtering out hidden messages

* Test against user being disabled instead of message visibility

* Fix automated test NRE

* Update comment

* Adjust federated engagement chat views. Closes #1617

* Add additional index to users table

* Add support for removing followers/requests. Closes #1630

* Reject federated actions from blocked actors. #1631

* Use fallback avatar if it fails to load. Closes #1635

* Fix styling of follower list. Closes #1636

* Add basic blurb stating they should follow the server. Closes #1641

* Update dev admin

* Set default go live message in migration. Closes #1642

* Reset the messages table on 0.0.11 schema migration

* Fix js error with moderation actions. Closes #1621

* Add a bit more clarification on follow modal. Closes #1599

* Remove todos

* Split out actor and domain blocking checks

* Check for errors on default values being set

* Clean up actor rejection due to being blocked

* Update dev admin

* Add colon to error to make it easier to read

* Remove markdown rendering of go live message. Reorganize text. Remove content warning. Closes #1645

* Break out the sort+render messages logic so it can be fired on visibility change. Closes #1643

* Do not send profile updates if federation is disabled

* Save follow references to inbound activities table

* Update dev admin

* Add blocked actor test

* Remove the overloaded term of Follow from social links

* Fix test running in memory only

* Remove "just" in engagement messags

* Replace star with heart for like action.

* Update dev admin

* Explicitly set cc as public

* Remove overly using the stream name in fediverse engagement messages

* Some federated/follow UI tweaks

* Remove explicit cc and bcc as they are not required

* Explicitly set the audience

* Remove extra margin

* Add Join Fediverse button to follow modal. Closes #1651

* Do not allow multiple follows to send multiple events. Closes #1650

* Give events a min height

* Do not allow old posts to be liked/shared. Closes #1652

* Remove value from log message

* Alert followers on private mode toggle

* Ignore clicks to follow button if disabled

* Remove underline from action buttons

* Add moderator icon to join message

* Update admin

* Post-merge remove unused var

* Remove pointing at feature branch

Co-authored-by: Ginger Wong <omqmail@gmail.com>
This commit is contained in:
Gabe Kangas
2022-01-12 13:53:10 -08:00
committed by GitHub
parent c51d9cdbf4
commit 045a0a2afd
174 changed files with 7295 additions and 404 deletions

View File

@@ -78,12 +78,39 @@ func SendSystemMessage(text string, ephemeral bool) error {
}
if !ephemeral {
saveEvent(message.ID, "system", message.Body, message.GetMessageType(), nil, message.Timestamp)
saveEvent(message.ID, nil, message.Body, message.GetMessageType(), nil, message.Timestamp, nil, nil, nil, nil)
}
return nil
}
// SendFediverseAction will send a message indicating some Fediverse engagement took place.
func SendFediverseAction(eventType string, userAccountName string, image *string, body string, link string) error {
message := events.FediverseEngagementEvent{
Event: events.Event{
Type: eventType,
},
MessageEvent: events.MessageEvent{
Body: body,
},
UserAccountName: userAccountName,
Image: image,
Link: link,
}
message.SetDefaults()
message.RenderBody()
if err := Broadcast(&message); err != nil {
log.Errorln("error sending system message", err)
return err
}
saveFederatedAction(message)
return nil
}
// SendSystemAction will send a system action string as an action event to all clients.
func SendSystemAction(text string, ephemeral bool) error {
message := events.ActionEvent{
@@ -100,7 +127,7 @@ func SendSystemAction(text string, ephemeral bool) error {
}
if !ephemeral {
saveEvent(message.ID, "action", message.Body, message.GetMessageType(), nil, message.Timestamp)
saveEvent(message.ID, nil, message.Body, message.GetMessageType(), nil, message.Timestamp, nil, nil, nil, nil)
}
return nil

View File

@@ -34,4 +34,10 @@ const (
ErrorMaxConnectionsExceeded EventType = "ERROR_MAX_CONNECTIONS_EXCEEDED"
// ErrorUserDisabled is an error returned when the connecting user has been previously banned/disabled.
ErrorUserDisabled EventType = "ERROR_USER_DISABLED"
// FediverseEngagementFollow is an event representing a follow action that took place on the fediverse.
FediverseEngagementFollow EventType = "FEDIVERSE_ENGAGEMENT_FOLLOW"
// FediverseEngagementLike is an event representing a like action that took place on the fediverse.
FediverseEngagementLike EventType = "FEDIVERSE_ENGAGEMENT_LIKE"
// FediverseEngagementRepost is an event representing a re-post action that took place on the fediverse.
FediverseEngagementRepost EventType = "FEDIVERSE_ENGAGEMENT_REPOST"
)

View File

@@ -0,0 +1,32 @@
package events
import "github.com/owncast/owncast/core/data"
// FediverseEngagementEvent is a message displayed in chat on representing an action on the Fediverse.
type FediverseEngagementEvent struct {
Event
MessageEvent
Image *string `json:"image"`
Link string `json:"link"`
UserAccountName string `json:"title"`
}
// GetBroadcastPayload will return the object to send to all chat users.
func (e *FediverseEngagementEvent) GetBroadcastPayload() EventPayload {
return EventPayload{
"id": e.ID,
"timestamp": e.Timestamp,
"body": e.Body,
"image": e.Image,
"type": e.Event.Type,
"title": e.UserAccountName,
"user": EventPayload{
"displayName": data.GetServerName(),
},
}
}
// GetMessageType will return the event type for this message.
func (e *FediverseEngagementEvent) GetMessageType() EventType {
return e.Event.Type
}

View File

@@ -34,17 +34,19 @@ func setupPersistence() {
// SaveUserMessage will save a single chat event to the messages database.
func SaveUserMessage(event events.UserMessageEvent) {
saveEvent(event.ID, event.User.ID, event.Body, event.Type, event.HiddenAt, event.Timestamp)
saveEvent(event.ID, &event.User.ID, event.Body, event.Type, event.HiddenAt, event.Timestamp, nil, nil, nil, nil)
}
func saveEvent(id string, userID string, body string, eventType string, hidden *time.Time, timestamp time.Time) {
func saveFederatedAction(event events.FediverseEngagementEvent) {
saveEvent(event.ID, nil, event.Body, event.Type, nil, event.Timestamp, event.Image, &event.Link, &event.UserAccountName, nil)
}
// nolint: unparam
func saveEvent(id string, userID *string, body string, eventType string, hidden *time.Time, timestamp time.Time, image *string, link *string, title *string, subtitle *string) {
defer func() {
_historyCache = nil
}()
_datastore.DbLock.Lock()
defer _datastore.DbLock.Unlock()
tx, err := _datastore.DB.Begin()
if err != nil {
log.Errorln("error saving", eventType, err)
@@ -53,7 +55,7 @@ func saveEvent(id string, userID string, body string, eventType string, hidden *
defer tx.Rollback() // nolint
stmt, err := tx.Prepare("INSERT INTO messages(id, user_id, body, eventType, hidden_at, timestamp) values(?, ?, ?, ?, ?, ?)")
stmt, err := tx.Prepare("INSERT INTO messages(id, user_id, body, eventType, hidden_at, timestamp, image, link, title, subtitle) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
if err != nil {
log.Errorln("error saving", eventType, err)
return
@@ -61,7 +63,7 @@ func saveEvent(id string, userID string, body string, eventType string, hidden *
defer stmt.Close()
if _, err = stmt.Exec(id, userID, body, eventType, hidden, timestamp); err != nil {
if _, err = stmt.Exec(id, userID, body, eventType, hidden, timestamp, image, link, title, subtitle); err != nil {
log.Errorln("error saving", eventType, err)
return
}
@@ -71,8 +73,134 @@ func saveEvent(id string, userID string, body string, eventType string, hidden *
}
}
func getChat(query string) []events.UserMessageEvent {
history := make([]events.UserMessageEvent, 0)
func makeUserMessageEventFromRowData(row rowData) events.UserMessageEvent {
scopes := ""
if row.userScopes != nil {
scopes = *row.userScopes
}
previousUsernames := []string{}
if row.previousUsernames != nil {
previousUsernames = strings.Split(*row.previousUsernames, ",")
}
displayName := ""
if row.userDisplayName != nil {
displayName = *row.userDisplayName
}
displayColor := 0
if row.userDisplayColor != nil {
displayColor = *row.userDisplayColor
}
createdAt := time.Time{}
if row.userCreatedAt != nil {
createdAt = *row.userCreatedAt
}
u := user.User{
ID: *row.userID,
AccessToken: "",
DisplayName: displayName,
DisplayColor: displayColor,
CreatedAt: createdAt,
DisabledAt: row.userDisabledAt,
NameChangedAt: row.userNameChangedAt,
PreviousNames: previousUsernames,
Scopes: strings.Split(scopes, ","),
}
message := events.UserMessageEvent{
Event: events.Event{
Type: row.eventType,
ID: row.id,
Timestamp: row.timestamp,
},
UserEvent: events.UserEvent{
User: &u,
HiddenAt: row.hiddenAt,
},
MessageEvent: events.MessageEvent{
Body: row.body,
RawBody: row.body,
},
}
return message
}
func makeSystemMessageChatEventFromRowData(row rowData) events.SystemMessageEvent {
message := events.SystemMessageEvent{
Event: events.Event{
Type: row.eventType,
ID: row.id,
Timestamp: row.timestamp,
},
MessageEvent: events.MessageEvent{
Body: row.body,
RawBody: row.body,
},
}
return message
}
func makeActionMessageChatEventFromRowData(row rowData) events.ActionEvent {
message := events.ActionEvent{
Event: events.Event{
Type: row.eventType,
ID: row.id,
Timestamp: row.timestamp,
},
MessageEvent: events.MessageEvent{
Body: row.body,
RawBody: row.body,
},
}
return message
}
func makeFederatedActionChatEventFromRowData(row rowData) events.FediverseEngagementEvent {
message := events.FediverseEngagementEvent{
Event: events.Event{
Type: row.eventType,
ID: row.id,
Timestamp: row.timestamp,
},
MessageEvent: events.MessageEvent{
Body: row.body,
RawBody: row.body,
},
Image: row.image,
Link: *row.link,
UserAccountName: *row.title,
}
return message
}
type rowData struct {
id string
userID *string
body string
eventType models.EventType
hiddenAt *time.Time
timestamp time.Time
title *string
subtitle *string
image *string
link *string
userDisplayName *string
userDisplayColor *int
userCreatedAt *time.Time
userDisabledAt *time.Time
previousUsernames *string
userNameChangedAt *time.Time
userScopes *string
}
func getChat(query string) []interface{} {
history := make([]interface{}, 0)
rows, err := _datastore.DB.Query(query)
if err != nil || rows.Err() != nil {
log.Errorln("error fetching chat history", err)
@@ -81,69 +209,47 @@ func getChat(query string) []events.UserMessageEvent {
defer rows.Close()
for rows.Next() {
var id string
var userID string
var body string
var messageType models.EventType
var hiddenAt *time.Time
var timestamp time.Time
var userDisplayName *string
var userDisplayColor *int
var userCreatedAt *time.Time
var userDisabledAt *time.Time
var previousUsernames *string
var userNameChangedAt *time.Time
row := rowData{}
// Convert a database row into a chat event
err = rows.Scan(&id, &userID, &body, &messageType, &hiddenAt, &timestamp, &userDisplayName, &userDisplayColor, &userCreatedAt, &userDisabledAt, &previousUsernames, &userNameChangedAt)
if err != nil {
if err = rows.Scan(
&row.id,
&row.userID,
&row.body,
&row.title,
&row.subtitle,
&row.image,
&row.link,
&row.eventType,
&row.hiddenAt,
&row.timestamp,
&row.userDisplayName,
&row.userDisplayColor,
&row.userCreatedAt,
&row.userDisabledAt,
&row.previousUsernames,
&row.userNameChangedAt,
&row.userScopes,
); err != nil {
log.Errorln("There is a problem converting query to chat objects. Please report this:", query)
break
}
// System messages and chat actions are special and are not from real users
if messageType == events.SystemMessageSent || messageType == events.ChatActionSent {
name := "Owncast"
userDisplayName = &name
color := 200
userDisplayColor = &color
}
var message interface{}
if previousUsernames == nil {
previousUsernames = userDisplayName
}
if userCreatedAt == nil {
now := time.Now()
userCreatedAt = &now
}
user := user.User{
ID: userID,
AccessToken: "",
DisplayName: *userDisplayName,
DisplayColor: *userDisplayColor,
CreatedAt: *userCreatedAt,
DisabledAt: userDisabledAt,
NameChangedAt: userNameChangedAt,
PreviousNames: strings.Split(*previousUsernames, ","),
}
message := events.UserMessageEvent{
Event: events.Event{
Type: messageType,
ID: id,
Timestamp: timestamp,
},
UserEvent: events.UserEvent{
User: &user,
HiddenAt: hiddenAt,
},
MessageEvent: events.MessageEvent{
Body: body,
RawBody: body,
},
switch row.eventType {
case events.MessageSent:
message = makeUserMessageEventFromRowData(row)
case events.SystemMessageSent:
message = makeSystemMessageChatEventFromRowData(row)
case events.ChatActionSent:
message = makeActionMessageChatEventFromRowData(row)
case events.FediverseEngagementFollow:
message = makeFederatedActionChatEventFromRowData(row)
case events.FediverseEngagementLike:
message = makeFederatedActionChatEventFromRowData(row)
case events.FediverseEngagementRepost:
message = makeFederatedActionChatEventFromRowData(row)
}
history = append(history, message)
@@ -152,16 +258,16 @@ func getChat(query string) []events.UserMessageEvent {
return history
}
var _historyCache *[]events.UserMessageEvent
var _historyCache *[]interface{}
// GetChatModerationHistory will return all the chat messages suitable for moderation purposes.
func GetChatModerationHistory() []events.UserMessageEvent {
func GetChatModerationHistory() []interface{} {
if _historyCache != nil {
return *_historyCache
}
// Get all messages regardless of visibility
query := "SELECT messages.id, user_id, body, eventType, hidden_at, timestamp, display_name, display_color, created_at, disabled_at, previous_names, namechanged_at FROM messages INNER JOIN users ON messages.user_id = users.id ORDER BY timestamp DESC"
query := "SELECT messages.id, user_id, body, title, subtitle, image, link, eventType, hidden_at, timestamp, display_name, display_color, created_at, disabled_at, previous_names, namechanged_at, scopes FROM messages INNER JOIN users ON messages.user_id = users.id ORDER BY timestamp DESC"
result := getChat(query)
_historyCache = &result
@@ -170,9 +276,9 @@ func GetChatModerationHistory() []events.UserMessageEvent {
}
// GetChatHistory will return all the chat messages suitable for returning as user-facing chat history.
func GetChatHistory() []events.UserMessageEvent {
func GetChatHistory() []interface{} {
// Get all visible messages
query := fmt.Sprintf("SELECT messages.id, user_id, body, eventType, hidden_at, timestamp, display_name, display_color, created_at, disabled_at, previous_names, namechanged_at FROM messages, users WHERE messages.user_id = users.id AND hidden_at IS NULL AND disabled_at IS NULL ORDER BY timestamp DESC LIMIT %d", maxBacklogNumber)
query := fmt.Sprintf("SELECT messages.id,messages.user_id, messages.body, messages.title, messages.subtitle, messages.image, messages.link, messages.eventType, messages.hidden_at, messages.timestamp, users.display_name, users.display_color, users.created_at, users.disabled_at, users.previous_names, users.namechanged_at, users.scopes FROM messages LEFT JOIN users ON messages.user_id = users.id WHERE hidden_at IS NULL AND disabled_at IS NULL ORDER BY timestamp DESC LIMIT %d", maxBacklogNumber)
m := getChat(query)
// Invert order of messages
@@ -200,7 +306,7 @@ func SetMessageVisibilityForUserID(userID string, visible bool) error {
}
for _, message := range messages {
ids = append(ids, message.ID)
ids = append(ids, message.(events.Event).ID)
}
// Tell the clients to hide/show these messages.
@@ -250,35 +356,3 @@ func saveMessageVisibility(messageIDs []string, visible bool) error {
return nil
}
// Only keep recent messages so we don't keep more chat data than needed
// for privacy and efficiency reasons.
func runPruner() {
_datastore.DbLock.Lock()
defer _datastore.DbLock.Unlock()
log.Traceln("Removing chat messages older than", maxBacklogHours, "hours")
deleteStatement := `DELETE FROM messages WHERE timestamp <= datetime('now', 'localtime', ?)`
tx, err := _datastore.DB.Begin()
if err != nil {
log.Debugln(err)
return
}
stmt, err := tx.Prepare(deleteStatement)
if err != nil {
log.Debugln(err)
return
}
defer stmt.Close()
if _, err = stmt.Exec(fmt.Sprintf("-%d hours", maxBacklogHours)); err != nil {
log.Debugln(err)
return
}
if err = tx.Commit(); err != nil {
log.Debugln(err)
return
}
}

39
core/chat/pruner.go Normal file
View File

@@ -0,0 +1,39 @@
package chat
import (
"fmt"
log "github.com/sirupsen/logrus"
)
// Only keep recent messages so we don't keep more chat data than needed
// for privacy and efficiency reasons.
func runPruner() {
_datastore.DbLock.Lock()
defer _datastore.DbLock.Unlock()
log.Traceln("Removing chat messages older than", maxBacklogHours, "hours")
deleteStatement := `DELETE FROM messages WHERE timestamp <= datetime('now', 'localtime', ?)`
tx, err := _datastore.DB.Begin()
if err != nil {
log.Debugln(err)
return
}
stmt, err := tx.Prepare(deleteStatement)
if err != nil {
log.Debugln(err)
return
}
defer stmt.Close()
if _, err = stmt.Exec(fmt.Sprintf("-%d hours", maxBacklogHours)); err != nil {
log.Debugln(err)
return
}
if err = tx.Commit(); err != nil {
log.Debugln(err)
return
}
}

13
core/data/activitypub.go Normal file
View File

@@ -0,0 +1,13 @@
package data
// GetFederatedInboxMap is a mapping between account names and their outbox.
func GetFederatedInboxMap() map[string]string {
return map[string]string{
GetDefaultFederationUsername(): GetDefaultFederationUsername(),
}
}
// GetDefaultFederationUsername will return the username used for sending federation activities.
func GetDefaultFederationUsername() string {
return GetFederationUsername()
}

View File

@@ -13,36 +13,47 @@ import (
log "github.com/sirupsen/logrus"
)
const extraContentKey = "extra_page_content"
const streamTitleKey = "stream_title"
const streamKeyKey = "stream_key"
const logoPathKey = "logo_path"
const serverSummaryKey = "server_summary"
const serverWelcomeMessageKey = "server_welcome_message"
const serverNameKey = "server_name"
const serverURLKey = "server_url"
const httpPortNumberKey = "http_port_number"
const httpListenAddressKey = "http_listen_address"
const rtmpPortNumberKey = "rtmp_port_number"
const serverMetadataTagsKey = "server_metadata_tags"
const directoryEnabledKey = "directory_enabled"
const directoryRegistrationKeyKey = "directory_registration_key"
const socialHandlesKey = "social_handles"
const peakViewersSessionKey = "peak_viewers_session"
const peakViewersOverallKey = "peak_viewers_overall"
const lastDisconnectTimeKey = "last_disconnect_time"
const ffmpegPathKey = "ffmpeg_path"
const nsfwKey = "nsfw"
const s3StorageEnabledKey = "s3_storage_enabled"
const s3StorageConfigKey = "s3_storage_config"
const videoLatencyLevel = "video_latency_level"
const videoStreamOutputVariantsKey = "video_stream_output_variants"
const chatDisabledKey = "chat_disabled"
const externalActionsKey = "external_actions"
const customStylesKey = "custom_styles"
const videoCodecKey = "video_codec"
const blockedUsernamesKey = "blocked_usernames"
const suggestedUsernamesKey = "suggested_usernames"
const (
extraContentKey = "extra_page_content"
streamTitleKey = "stream_title"
streamKeyKey = "stream_key"
logoPathKey = "logo_path"
serverSummaryKey = "server_summary"
serverWelcomeMessageKey = "server_welcome_message"
serverNameKey = "server_name"
serverURLKey = "server_url"
httpPortNumberKey = "http_port_number"
httpListenAddressKey = "http_listen_address"
rtmpPortNumberKey = "rtmp_port_number"
serverMetadataTagsKey = "server_metadata_tags"
directoryEnabledKey = "directory_enabled"
directoryRegistrationKeyKey = "directory_registration_key"
socialHandlesKey = "social_handles"
peakViewersSessionKey = "peak_viewers_session"
peakViewersOverallKey = "peak_viewers_overall"
lastDisconnectTimeKey = "last_disconnect_time"
ffmpegPathKey = "ffmpeg_path"
nsfwKey = "nsfw"
s3StorageEnabledKey = "s3_storage_enabled"
s3StorageConfigKey = "s3_storage_config"
videoLatencyLevel = "video_latency_level"
videoStreamOutputVariantsKey = "video_stream_output_variants"
chatDisabledKey = "chat_disabled"
externalActionsKey = "external_actions"
customStylesKey = "custom_styles"
videoCodecKey = "video_codec"
blockedUsernamesKey = "blocked_usernames"
publicKeyKey = "public_key"
privateKeyKey = "private_key"
serverInitDateKey = "server_init_date"
federationEnabledKey = "federation_enabled"
federationUsernameKey = "federation_username"
federationPrivateKey = "federation_private"
federationGoLiveMessageKey = "federation_go_live_message"
federationShowEngagementKey = "federation_show_engagement"
federationBlockedDomainsKey = "federation_blocked_domains"
suggestedUsernamesKey = "suggested_usernames"
)
// GetExtraPageBodyContent will return the user-supplied body content.
func GetExtraPageBodyContent() string {
@@ -295,7 +306,7 @@ func GetSocialHandles() []models.SocialHandle {
// SetSocialHandles will set the external social links.
func SetSocialHandles(socialHandles []models.SocialHandle) error {
var configEntry = ConfigEntry{Key: socialHandlesKey, Value: socialHandles}
configEntry := ConfigEntry{Key: socialHandlesKey, Value: socialHandles}
return _datastore.Save(configEntry)
}
@@ -350,7 +361,7 @@ func GetLastDisconnectTime() (*utils.NullTime, error) {
// SetLastDisconnectTime will set the time the last stream ended.
func SetLastDisconnectTime(disconnectTime time.Time) error {
savedDisconnectTime := utils.NullTime{Time: disconnectTime, Valid: true}
var configEntry = ConfigEntry{Key: lastDisconnectTimeKey, Value: savedDisconnectTime}
configEntry := ConfigEntry{Key: lastDisconnectTimeKey, Value: savedDisconnectTime}
return _datastore.Save(configEntry)
}
@@ -399,7 +410,7 @@ func GetS3Config() models.S3 {
// SetS3Config will set the external storage configuration.
func SetS3Config(config models.S3) error {
var configEntry = ConfigEntry{Key: s3StorageConfigKey, Value: config}
configEntry := ConfigEntry{Key: s3StorageConfigKey, Value: config}
return _datastore.Save(configEntry)
}
@@ -457,7 +468,7 @@ func GetStreamOutputVariants() []models.StreamOutputVariant {
// SetStreamOutputVariants will set the stream output variants.
func SetStreamOutputVariants(variants []models.StreamOutputVariant) error {
var configEntry = ConfigEntry{Key: videoStreamOutputVariantsKey, Value: variants}
configEntry := ConfigEntry{Key: videoStreamOutputVariantsKey, Value: variants}
return _datastore.Save(configEntry)
}
@@ -493,7 +504,7 @@ func GetExternalActions() []models.ExternalAction {
// SetExternalActions will save external actions.
func SetExternalActions(actions []models.ExternalAction) error {
var configEntry = ConfigEntry{Key: externalActionsKey, Value: actions}
configEntry := ConfigEntry{Key: externalActionsKey, Value: actions}
return _datastore.Save(configEntry)
}
@@ -583,7 +594,6 @@ func FindHighestVideoQualityIndex(qualities []models.StreamOutputVariant) int {
// GetForbiddenUsernameList will return the blocked usernames as a comma separated string.
func GetForbiddenUsernameList() []string {
usernameString, err := _datastore.GetString(blockedUsernamesKey)
if err != nil {
return config.DefaultForbiddenUsernames
}
@@ -622,3 +632,125 @@ func SetSuggestedUsernamesList(usernames []string) error {
usernameListString := strings.Join(usernames, ",")
return _datastore.SetString(suggestedUsernamesKey, usernameListString)
}
// GetServerInitTime will return when the server was first setup.
func GetServerInitTime() (*utils.NullTime, error) {
var t utils.NullTime
configEntry, err := _datastore.Get(serverInitDateKey)
if err != nil {
return nil, err
}
if err := configEntry.getObject(&t); err != nil {
return nil, err
}
if !t.Valid {
return nil, err
}
return &t, nil
}
// SetServerInitTime will set when the server was first created.
func SetServerInitTime(t time.Time) error {
nt := utils.NullTime{Time: t, Valid: true}
configEntry := ConfigEntry{Key: serverInitDateKey, Value: nt}
return _datastore.Save(configEntry)
}
// SetFederationEnabled will enable federation if set to true.
func SetFederationEnabled(enabled bool) error {
return _datastore.SetBool(federationEnabledKey, enabled)
}
// GetFederationEnabled will return if federation is enabled.
func GetFederationEnabled() bool {
enabled, err := _datastore.GetBool(federationEnabledKey)
if err == nil {
return enabled
}
return false
}
// SetFederationUsername will set the username used in federated activities.
func SetFederationUsername(username string) error {
return _datastore.SetString(federationUsernameKey, username)
}
// GetFederationUsername will return the username used in federated activities.
func GetFederationUsername() string {
username, err := _datastore.GetString(federationUsernameKey)
if username == "" || err != nil {
return config.GetDefaults().FederationUsername
}
return username
}
// SetFederationGoLiveMessage will set the message sent when going live.
func SetFederationGoLiveMessage(message string) error {
return _datastore.SetString(federationGoLiveMessageKey, message)
}
// GetFederationGoLiveMessage will return the message sent when going live.
func GetFederationGoLiveMessage() string {
// Empty message means it's disabled.
message, err := _datastore.GetString(federationGoLiveMessageKey)
if err != nil {
log.Traceln("unable to fetch go live message.", err)
}
return message
}
// SetFederationIsPrivate will set if federation activity is private.
func SetFederationIsPrivate(isPrivate bool) error {
return _datastore.SetBool(federationPrivateKey, isPrivate)
}
// GetFederationIsPrivate will return if federation is private.
func GetFederationIsPrivate() bool {
isPrivate, err := _datastore.GetBool(federationPrivateKey)
if err == nil {
return isPrivate
}
return false
}
// SetFederationShowEngagement will set if fediverse engagement shows in chat.
func SetFederationShowEngagement(showEngagement bool) error {
return _datastore.SetBool(federationShowEngagementKey, showEngagement)
}
// GetFederationShowEngagement will return if fediverse engagement shows in chat.
func GetFederationShowEngagement() bool {
showEngagement, err := _datastore.GetBool(federationShowEngagementKey)
if err == nil {
return showEngagement
}
return true
}
// SetBlockedFederatedDomains will set the blocked federated domains.
func SetBlockedFederatedDomains(domains []string) error {
return _datastore.SetString(federationBlockedDomainsKey, strings.Join(domains, ","))
}
// GetBlockedFederatedDomains will return a list of blocked federated domains.
func GetBlockedFederatedDomains() []string {
domains, err := _datastore.GetString(federationBlockedDomainsKey)
if err != nil {
return []string{}
}
if domains == "" {
return []string{}
}
return strings.Split(domains, ",")
}

23
core/data/crypto.go Normal file
View File

@@ -0,0 +1,23 @@
package data
// GetPublicKey will return the public key.
func GetPublicKey() string {
value, _ := _datastore.GetString(publicKeyKey)
return value
}
// SetPublicKey will save the public key.
func SetPublicKey(key string) error {
return _datastore.SetString(publicKeyKey, key)
}
// GetPrivateKey will return the private key.
func GetPrivateKey() string {
value, _ := _datastore.GetString(privateKeyKey)
return value
}
// SetPrivateKey will save the private key.
func SetPrivateKey(key string) error {
return _datastore.SetString(privateKeyKey, key)
}

View File

@@ -17,11 +17,13 @@ import (
)
const (
schemaVersion = 1
schemaVersion = 3
)
var _db *sql.DB
var _datastore *Datastore
var (
_db *sql.DB
_datastore *Datastore
)
// GetDatabase will return the shared instance of the actual database.
func GetDatabase() *sql.DB {
@@ -35,18 +37,34 @@ func GetStore() *Datastore {
// SetupPersistence will open the datastore and make it available.
func SetupPersistence(file string) error {
// Create empty DB file if it doesn't exist.
if !utils.DoesFileExists(file) {
log.Traceln("Creating new database at", file)
// Allow support for in-memory databases for tests.
_, err := os.Create(file)
var db *sql.DB
if file == ":memory:" {
inMemoryDb, err := sql.Open("sqlite3", file)
if err != nil {
log.Fatal(err.Error())
}
}
db = inMemoryDb
} else {
// Create empty DB file if it doesn't exist.
if !utils.DoesFileExists(file) {
log.Traceln("Creating new database at", file)
db, err := sql.Open("sqlite3", fmt.Sprintf("file:%s?_cache_size=10000&cache=shared&_journal_mode=WAL", file))
db.SetMaxOpenConns(1)
_, err := os.Create(file)
if err != nil {
log.Fatal(err.Error())
}
}
onDiskDb, err := sql.Open("sqlite3", fmt.Sprintf("file:%s?_cache_size=10000&cache=shared&_journal_mode=WAL", file))
if err != nil {
return err
}
db = onDiskDb
db.SetMaxOpenConns(1)
}
_db = db
// Some SQLite optimizations
@@ -58,10 +76,6 @@ func SetupPersistence(file string) error {
createWebhooksTable()
createUsersTable(db)
if err != nil {
return err
}
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS config (
"key" string NOT NULL PRIMARY KEY,
"value" TEXT
@@ -70,7 +84,7 @@ func SetupPersistence(file string) error {
}
var version int
err = db.QueryRow("SELECT value FROM config WHERE key='version'").
err := db.QueryRow("SELECT value FROM config WHERE key='version'").
Scan(&version)
if err != nil {
if err != sql.ErrNoRows {
@@ -117,10 +131,14 @@ func migrateDatabase(db *sql.DB, from, to int) error {
dbBackupFile := filepath.Join(config.BackupDirectory, fmt.Sprintf("owncast-v%d.bak", from))
utils.Backup(db, dbBackupFile)
for v := from; v < to; v++ {
log.Tracef("Migration step from %d to %d\n", v, v+1)
switch v {
case 0:
log.Tracef("Migration step from %d to %d\n", v, v+1)
migrateToSchema1(db)
case 1:
migrateToSchema2(db)
case 2:
migrateToSchema3(db)
default:
log.Fatalln("missing database migration step")
}

View File

@@ -14,6 +14,14 @@ func HasPopulatedDefaults() bool {
return hasPopulated
}
func hasPopulatedFederationDefaults() bool {
hasPopulated, err := _datastore.GetBool("HAS_POPULATED_FEDERATION_DEFAULTS")
if err != nil {
return false
}
return hasPopulated
}
// PopulateDefaults will set default values in the database.
func PopulateDefaults() {
defaults := config.GetDefaults()
@@ -32,6 +40,7 @@ func PopulateDefaults() {
_ = SetServerName("Owncast")
_ = SetStreamKey(defaults.StreamKey)
_ = SetExtraPageBodyContent("This is your page's content that can be edited in the admin.")
_ = SetFederationGoLiveMessage(defaults.FederationGoLiveMessage)
_ = SetSocialHandles([]models.SocialHandle{
{
Platform: "github",

View File

@@ -15,6 +15,10 @@ func CreateMessagesTable(db *sql.DB) {
"eventType" TEXT,
"hidden_at" DATETIME,
"timestamp" DATETIME,
"title" TEXT,
"subtitle" TEXT,
"image" TEXT,
"link" TEXT,
PRIMARY KEY (id)
);CREATE INDEX index ON messages (id, user_id, hidden_at, timestamp);
CREATE INDEX id ON messages (id);

View File

@@ -9,6 +9,44 @@ import (
"github.com/teris-io/shortid"
)
func migrateToSchema3(db *sql.DB) {
// Since it's just a backlog of chat messages let's wipe the old messages
// and recreate the table.
// Drop the old messages table
stmt, err := db.Prepare("DROP TABLE messages")
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
_, err = stmt.Exec()
if err != nil {
log.Warnln(err)
}
// Recreate it
CreateMessagesTable(db)
}
func migrateToSchema2(db *sql.DB) {
// Since it's just a backlog of chat messages let's wipe the old messages
// and recreate the table.
// Drop the old messages table
stmt, err := db.Prepare("DROP TABLE messages")
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
_, err = stmt.Exec()
if err != nil {
log.Warnln(err)
}
// Recreate it
CreateMessagesTable(db)
}
func migrateToSchema1(db *sql.DB) {
// Since it's just a backlog of chat messages let's wipe the old messages
// and recreate the table.
@@ -100,7 +138,6 @@ func insertAPIToken(db *sql.DB, token string, name string, color int, scopes str
return err
}
stmt, err := tx.Prepare("INSERT INTO users(id, access_token, display_name, display_color, scopes, type) values(?, ?, ?, ?, ?, ?)")
if err != nil {
return err
}

View File

@@ -5,9 +5,12 @@ import (
"database/sql"
"encoding/gob"
"sync"
"time"
// sqlite requires a blank import.
_ "github.com/mattn/go-sqlite3"
"github.com/owncast/owncast/config"
"github.com/owncast/owncast/db"
log "github.com/sirupsen/logrus"
)
@@ -37,6 +40,11 @@ func (ds *Datastore) warmCache() {
}
}
// GetQueries will return the shared instance of the SQL query generator.
func (ds *Datastore) GetQueries() *db.Queries {
return db.New(ds.DB)
}
// Get will query the database for the key and return the entry.
func (ds *Datastore) Get(key string) (ConfigEntry, error) {
cachedValue, err := ds.GetCachedValue(key)
@@ -125,6 +133,20 @@ func (ds *Datastore) Setup() {
if !HasPopulatedDefaults() {
PopulateDefaults()
}
if !hasPopulatedFederationDefaults() {
if err := SetFederationGoLiveMessage(config.GetDefaults().FederationGoLiveMessage); err != nil {
log.Errorln(err)
}
if err := _datastore.SetBool("HAS_POPULATED_FEDERATION_DEFAULTS", true); err != nil {
log.Errorln(err)
}
}
// Set the server initialization date if needed.
if hasSetInitDate, _ := GetServerInitTime(); hasSetInitDate == nil || !hasSetInitDate.Valid {
_ = SetServerInitTime(time.Now())
}
}
// Reset will delete all config entries in the datastore and start over.

View File

@@ -24,6 +24,7 @@ func createUsersTable(db *sql.DB) {
PRIMARY KEY (id)
);CREATE INDEX index ON users (id, access_token, disabled_at);
CREATE INDEX id ON users (id);
CREATE INDEX id_disabled ON users (id, disabled_at);
CREATE INDEX access_token ON users (access_token);
CREATE INDEX disabled_at ON USERS (disabled_at);`

View File

@@ -1,11 +1,13 @@
package core
import (
"context"
"io"
"time"
log "github.com/sirupsen/logrus"
"github.com/owncast/owncast/activitypub"
"github.com/owncast/owncast/config"
"github.com/owncast/owncast/core/chat"
"github.com/owncast/owncast/core/data"
@@ -24,6 +26,8 @@ var _onlineCleanupTicker *time.Ticker
var _currentBroadcast *models.CurrentBroadcast
var _onlineTimerCancelFunc context.CancelFunc
// setStreamAsConnected sets the stream as connected.
func setStreamAsConnected(rtmpOut *io.PipeReader) {
now := utils.NullTime{Time: time.Now(), Valid: true}
@@ -66,6 +70,11 @@ func setStreamAsConnected(rtmpOut *io.PipeReader) {
_ = chat.SendSystemAction("Stay tuned, the stream is **starting**!", true)
chat.SendAllWelcomeMessage()
// Send a delayed live Federated message.
if data.GetFederationEnabled() {
_onlineTimerCancelFunc = startFederatedLiveStreamMessageTimer()
}
}
// SetStreamAsDisconnected sets the stream as disconnected.
@@ -73,6 +82,10 @@ func SetStreamAsDisconnected() {
_ = chat.SendSystemAction("The stream is ending.", true)
now := utils.NullTime{Time: time.Now(), Valid: true}
if _onlineTimerCancelFunc != nil {
_onlineTimerCancelFunc()
}
_stats.StreamConnected = false
_stats.LastDisconnectTime = &now
_stats.LastConnectTime = nil
@@ -147,3 +160,20 @@ func stopOnlineCleanupTimer() {
_onlineCleanupTicker.Stop()
}
}
func startFederatedLiveStreamMessageTimer() context.CancelFunc {
// Send a delayed live Federated message.
c, cancelFunc := context.WithCancel(context.Background())
_onlineTimerCancelFunc = cancelFunc
go func(c context.Context) {
select {
case <-time.After(time.Minute * 2.0):
log.Traceln("Sending Federated Go Live message.")
if err := activitypub.SendLive(); err != nil {
log.Errorln(err)
}
case <-c.Done():
}
}(c)
return cancelFunc
}

View File

@@ -110,10 +110,7 @@ func fireThumbnailGenerator(segmentPath string, variantIndex int) error {
log.Errorln(err)
}
// If YP support is enabled also create an animated GIF preview
if data.GetDirectoryEnabled() {
makeAnimatedGifPreview(mostRecentFile, previewGifFile)
}
makeAnimatedGifPreview(mostRecentFile, previewGifFile)
return nil
}