Refactored stream observations and config refresh handling.
CI / Formatting (push) Successful in 6s
CI / Linting (push) Successful in 6s
CI / Tests (push) Successful in 27s
CI / Type Checking (push) Successful in 23s
CI / Spelling (push) Successful in 19s

This commit is contained in:
2026-05-21 19:44:29 -04:00
parent c837b916c0
commit 38e91add32
14 changed files with 1503 additions and 512 deletions
+390
View File
@@ -0,0 +1,390 @@
# OwncastSentry Package
This package contains the Maubot plugin runtime code for OwncastSentry. It
tracks room subscriptions to Owncast instances, polls subscribed streams,
detects stream state changes, and sends Matrix notifications.
This README gives maintainers a high-level map of how the package fits
together and where the important behavior lives. User-facing setup and command
usage belong outside this package. Function-level details belong in docstrings
or tests.
## Files
- `__init__.py`: Maubot plugin entry point, service wiring, command
decorators, scheduler loop, shutdown, and metrics endpoint.
- `config.py`: Maubot configuration proxy for the Prometheus metrics endpoint
setting.
- `types.py`: immutable value objects, Owncast API response validation, derived
stream status, update results, and subscription domain errors.
- `repository.py`: database schema migrations plus stream and subscription
repositories.
- `owncast_client.py`: bounded aiohttp client for Owncast `/api/status` and
`/api/config` requests.
- `subscription_manager.py`: subscription use cases, domain normalization, and
first-subscription Owncast validation.
- `stream_monitor.py`: stream update orchestration, state transition detection,
progressive failure backoff, cleanup thresholds, notification decisions, and
stream metrics.
- `notification_service.py`: Matrix notification formatting, broadcast,
per-domain notification cooldowns, cleanup notices, and delivery metrics.
- `commands.py`: Maubot command handlers for subscribe, unsubscribe,
subscriptions, and live listings.
- `metrics.py`: isolated Prometheus registry, counters, gauges, and response
timing helpers.
The package has a service-layer shape: `OwncastSentry.start()` builds the
services, commands call `CommandHandler`, command handlers call
`SubscriptionManager`, scheduled updates call `StreamMonitor`, and
`StreamMonitor` coordinates repositories, the Owncast client, notifications,
and metrics.
```mermaid
flowchart LR
Maubot["Maubot loads OwncastSentry"] --> Start["OwncastSentry.start()"]
Start --> Repos["StreamRepository and SubscriptionRepository"]
Start --> Owncast["OwncastClient"]
Start --> Metrics["MetricsService"]
Start --> Notify["NotificationService"]
Start --> Monitor["StreamMonitor"]
Start --> Manager["SubscriptionManager"]
Start --> Commands["CommandHandler"]
Commands --> Manager
Manager --> Repos
Manager --> Owncast
Monitor --> Repos
Monitor --> Owncast
Monitor --> Notify
Monitor --> Metrics
Notify --> Repos
Notify --> Matrix["Matrix client"]
```
## Core Concepts
A few terms are used throughout the package:
- A stream is a normalized Owncast domain tracked by the plugin. The database
stores one stream row per domain, even if multiple Matrix rooms subscribe to
it.
- A subscription is a Matrix room's request to receive notifications for one
stream domain.
- `StreamStateObservation` is one successful Owncast status API sample. The
monitor decides whether the observation starts a new persisted status period.
- `StreamConfigObservation` is metadata fetched from Owncast's config API,
currently the instance name and tags used in notifications and periodic
display-name refreshes.
- `StreamState` is an immutable snapshot of the latest persisted state for a
stream: display metadata, the latest Owncast online flag, when that status was
first observed by the bot, failure count, and the derived online, offline, or
unknown status.
Durable state lives in the database. The `streams` table stores the latest
known display and state fields for each Owncast domain, and the `subscriptions`
table stores which Matrix rooms follow each domain.
Subscriptions are room-scoped, while stream state is shared per domain.
Notification delivery fans out from one stored stream state to the rooms
currently subscribed to that domain.
Stream identity is the normalized bare domain. Repositories, metrics,
notification state, and Owncast API calls all use that domain rather than the
original user-supplied URL.
Failures stay isolated at plugin boundaries. A bad command, failed stream
check, or room delivery is logged and counted without stopping unrelated
commands, stream updates, or notifications.
`StreamStateObservation`, `StreamConfigObservation`, `StreamState`,
`UpdateResult`, and `RoomSubscription` are immutable snapshots. State changes
are represented by new values that are persisted through repositories instead of
mutating existing objects.
## Service Wiring
`OwncastSentry.start()` is the composition root for one plugin instance. It
loads config, creates the metrics service, Owncast API client, repositories,
notification service, stream monitor, subscription manager, and command handler,
then passes dependencies into each service explicitly.
This keeps lower-level modules independent of Maubot globals. Commands do not
construct repositories, `StreamMonitor` does not know about the Maubot
scheduler, and notification delivery does not fetch its own subscription state
from the plugin instance.
The plugin boundary owns scheduling. The minute loop loads subscribed domains
from `SubscriptionRepository` and calls `StreamMonitor.update_all_streams()`.
The same pattern applies to new background work: schedule it from the plugin
boundary, keep the worker service callable directly, and close owned external
resources from `OwncastSentry.stop()`.
## Subscription Management
`SubscriptionManager` coordinates the room-to-domain relationship. Command
handlers pass it user-supplied stream targets, and it turns those targets into
normalized domains before touching repository state.
Subscribing a room creates a subscription for the normalized domain and creates
the shared stream row when the domain is first seen. First-time domains are
validated through `OwncastClient`; domains that already have subscribers reuse
the existing stream record instead of revalidating.
Unsubscribing removes one room's subscription to a domain. It does not delete
the shared stream row or make remote Owncast requests. Long-term dead stream
cleanup is owned by `StreamMonitor`.
Listing methods stay room-scoped. They return the subscriptions relevant to one
Matrix room, with the shared stream state attached for display.
## Owncast API Client
`OwncastClient` owns the aiohttp session used for Owncast API requests. It
fetches stream status and instance config, validates response shape, and turns
accepted responses into the value objects used by the rest of the package.
The session uses a plugin-specific User-Agent, a dummy cookie jar, no DNS cache,
a global connection limit, a per-host connection limit of one, and connect/read
socket timeouts. aiohttp's own DNS cache is disabled so DNS caching and policy
stay with the resolver configured for the deployment.
The per-host connection limit and keepalive settings are intentional. Each
Owncast instance gets at most one reusable connection, so minute-by-minute
status checks can reuse the existing TLS session instead of opening a fresh
connection for every poll.
Responses are accepted only when they are HTTP 200 JSON objects under the
configured size limit. Invalid JSON, non-object JSON, oversized bodies,
unexpected status codes, malformed API fields, connection errors, and timeouts
are logged and ignored.
The response-time metric is recorded only when a request completes and the
parsed response shape is valid. Failed requests remove any stale timing label
for that domain.
## Stream Monitoring
`StreamMonitor` owns the core per-domain state machine. `update_stream()`
compares the stored stream state with the latest Owncast status response,
decides whether to send a notification, updates persistent state when needed,
and records stream metrics. Each poll first handles missing rows, polling
backoff, and failed Owncast requests. Only a valid status response enters the
stream transition policy. `update_all_streams()` wraps this flow for many
domains, isolates per-domain exceptions, and records subscription-count metrics.
```mermaid
flowchart TD
Start["update_stream(domain)"] --> Old["load stored StreamState"]
Old --> Missing{"row exists?"}
Missing -->|no| Noop["return success"]
Missing -->|yes| Backoff{"backoff allows poll?"}
Backoff -->|no| Skip["skip HTTP call<br/>increment failure counter"]
Skip --> CleanupSkip["run warning/deletion cleanup checks"]
CleanupSkip --> MetricsSkip["if row remains, update<br/>status and failure metrics"]
MetricsSkip --> SuccessSkip["return success"]
Backoff -->|yes| Fetch["fetch and validate /api/status"]
Fetch --> Fetched{"got valid state?"}
Fetched -->|no| Failure["increment failure counter"]
Failure --> CleanupFail["run warning/deletion cleanup checks"]
CleanupFail --> MetricsFail["if row remains, update<br/>status and failure metrics"]
MetricsFail --> Failed["return failure"]
Fetched -->|yes| Reset["reset failure counter and metric"]
Reset --> Timer["ensure offline timer<br/>cache entry exists"]
Timer --> Transition["classify stream transition"]
Transition --> First{"first observation?"}
First -->|yes| FirstLog["log suppressed notification"]
First -->|no| LiveCheck{"went live?"}
FirstLog --> ConfigDecision
LiveCheck -->|yes| Brief{"observed offline less than<br/>temporary cooldown?"}
LiveCheck -->|no| TitleChanged{"title changed<br/>while online?"}
Brief -->|no| LiveNotify["go-live notification needed"]
Brief -->|yes| BriefTitle{"title changed?"}
BriefTitle -->|yes| BriefNotify["title-change notification needed"]
BriefTitle -->|no| NoNotify["no notification"]
TitleChanged -->|yes| OfflineMarker{"offline marker newer than<br/>last notification?"}
TitleChanged -->|no| WentOffline{"went offline?"}
WentOffline -->|yes| Offline["record offline timer"]
WentOffline -->|no| NoNotify
OfflineMarker -->|yes| ResumeNotify["go-live notification needed"]
OfflineMarker -->|no| TitleNotify["title-change notification needed"]
NoNotify --> ConfigDecision
Offline --> ConfigDecision
LiveNotify --> ConfigDecision
BriefNotify --> ConfigDecision
ResumeNotify --> ConfigDecision
TitleNotify --> ConfigDecision
ConfigDecision{"notification, first observation,<br/>or hourly refresh?"}
ConfigDecision -->|yes| Config["fetch /api/config"]
ConfigDecision -->|no| NotifyGate{"notification needed?"}
Config --> NotifyGate
NotifyGate -->|yes| Send["attempt live/title notification"]
NotifyGate -->|no| Save["save stream row<br/>repository ignores unchanged values"]
Send --> Save
Save --> FinalMetrics["set current status metric"]
FinalMetrics --> Done["return success"]
```
Failure counters drive both status and polling behavior. A stream is displayed
as unknown when its failure counter is above `UNKNOWN_STATUS_THRESHOLD`.
Counters 0 through 4 poll every minute; later tiers poll only on selected
counter values and eventually only every fifteenth minute. Skipped cycles still
increment the counter so cleanup can progress.
Transition decisions use Owncast's `online` field. The stream status timestamp is
the bot's local UTC observation time for the first successful poll, a missing
timestamp repair, or an online/offline transition. Owncast's remote connect,
disconnect, and server timestamps are not required for status parsing.
First observation is special. If a new stream row has no status timestamp, the
first successful update fetches config and stores state but suppresses
notification attempts, even if the stream is already live.
Go-live notification attempts happen when a stream moves from offline to online
and the monitor's in-memory offline timer shows it was not merely a brief
outage. A stream that returns within the temporary offline cooldown only attempts
a notification when the title changed, and then it uses a title change
notification.
Mid-session title changes attempt title change notifications. If the monitor has
an offline marker that is newer than the last notification, it attempts a normal
live notification instead so rooms do not see a title change for what is
effectively a new session.
Offline transitions do not notify rooms. They update the database and record a
monotonic offline timestamp used by later live/title notification policy.
The live/title notification send step is an attempt, not a guaranteed Matrix message.
`NotificationService` still applies the shared live/title cooldown and records
the cooldown only when at least one room receives the message.
`/api/config` is fetched when the monitor is about to attempt a live or
title-change notification, on a stream's first successful observation, and
during the hourly refresh window for successfully polled streams. The config
response supplies the display name and tags for notifications. If config is
fetched successfully during an update, the persisted stream name is refreshed
from it; otherwise the stored name is left unchanged.
## Cleanup Policy
A domain that remains unreachable is eventually cleaned up. The failure counter
represents minute ticks, including skipped backoff ticks.
At the 83-day threshold, `StreamMonitor` sends a cleanup warning to all rooms
subscribed to the domain. At the 90-day threshold or beyond, it sends a deletion
notice, deletes all subscriptions for the domain, deletes the stream row, clears
the monitor and notification caches for that domain, and removes per-domain
metrics.
```mermaid
flowchart TD
Counter["failure counter after failed or skipped check"] --> Warn{"counter == 83 days?"}
Warn -->|yes| Warning["send cleanup warning"]
Warn -->|no| Delete
Warning --> Delete{"counter >= 90 days?"}
Delete -->|no| Done["done"]
Delete -->|yes| Notice["send cleanup deletion notice"]
Notice --> Subs["delete subscriptions"]
Subs --> Stream["delete stream row"]
Stream --> Caches["clear local caches"]
Caches --> Metrics["remove per-domain metrics"]
Metrics --> DoneCleanup["done"]
```
## Notification Delivery
`NotificationService` sends plain text Matrix messages to every room subscribed
to a domain. Broadcasts run concurrently and use `asyncio.gather(...,
return_exceptions=True)` so one failed room does not block delivery to other
rooms.
Live and title-change notifications share a per-domain cooldown. The cooldown
is recorded only when at least one room receives the message. If there are no
subscribed rooms, or every delivery fails, no cooldown is recorded.
Notification text is whitespace-normalized before sending. Stream names fall
back to the domain when config lookup fails or the instance name is empty. Tags
are appended as hash tags after filtering out empty tags and tags that start
with a dot.
Cleanup warning and deletion notices bypass the live/title cooldown, but still
record delivery metrics.
## Command Handling
`OwncastSentry` exposes Maubot command decorators in `__init__.py`, but command
behavior lives in `CommandHandler`. The plugin entry points are thin wrappers
that catch unexpected exceptions and record command error metrics.
`CommandHandler.subscribe()` and `CommandHandler.unsubscribe()` translate
domain errors into user-facing Matrix replies. `subscriptions()` and `live()`
read resolved room subscriptions and render Markdown output with escaped
untrusted text.
Command listings are room-scoped and ordered by domain through repository
queries. `subscriptions()` includes online, offline, and unknown streams.
`live()` includes only rows marked online with a failure count at or below the
unknown threshold.
## Persistence
The database schema is managed by `repository.py` through Maubot's upgrade
table.
- Revision 1 creates `streams` and `subscriptions`.
- Revision 2 fixes `subscriptions.stream_domain` from `INTEGER` to `TEXT`.
- Revision 3 adds `streams.failure_counter`.
- Revision 4 replaces separate connect/disconnect timestamps with
`streams.online` and `streams.status_since`.
Repositories own SQL access. `StreamRepository.update()` writes display
and state fields; failure counters use dedicated methods.
`SubscriptionRepository` raises domain-specific errors for duplicate adds and
missing removes.
Resolved subscription listings join `subscriptions` to `streams`, which means
orphaned subscription rows without a stream row are skipped in room display
queries.
## Metrics
`MetricsService` uses a private `CollectorRegistry` so plugin metrics do not mix
with process-global Prometheus collectors.
The registered metrics cover:
- notification delivery attempts by type and result;
- current stream status per domain (`1` online, `0` offline, `-1` unknown);
- subscription count per domain;
- consecutive check failures per domain;
- last successful Owncast API response duration per domain;
- build/version information;
- open aiohttp connection count;
- internal scheduler and command errors.
Known counter label combinations are initialized to zero. This keeps the
Prometheus output stable before any events have occurred.
## Test Map
Start with these tests when changing package behavior:
- `tests/test_types.py`: immutable value objects, API response shape checks,
truncation, stream status derivation, and subscription errors.
- `tests/test_repository.py`: schema-backed stream and subscription repository
behavior.
- `tests/test_subscription_manager.py`: domain normalization, subscribe and
unsubscribe workflows, validation skipping, and room listing delegation.
- `tests/test_owncast_client.py`: Owncast API request handling, response body
limits, validation failures, User-Agent, response timing, and connection
counts.
- `tests/test_stream_monitor.py`: polling backoff, first-update suppression,
live/offline/title transitions, cleanup thresholds, exception isolation, and
monitor metrics.
- `tests/test_notification_service.py`: message formatting, sanitization,
cooldown behavior, broadcast failure accounting, cleanup notices, and
notification metrics.
- `tests/test_commands.py`: end-to-end Maubot command behavior, Markdown
escaping, duration formatting, room-scoped subscription listings, and live
listings.
- `tests/test_metrics.py`: Prometheus counters, gauges, response timer,
per-domain label removal, build info, open connections, and isolated registry
output.
+1 -1
View File
@@ -78,7 +78,7 @@ def _format_duration(timestamp_str: str, now: datetime) -> str:
if seconds < _SECONDS_PER_DAY:
hours = seconds // _SECONDS_PER_HOUR
return f"{hours} hour{'s' if hours != 1 else ''}"
except (TypeError, ValueError):
except TypeError, ValueError:
return "unknown duration"
else:
days = seconds // _SECONDS_PER_DAY
+3 -3
View File
@@ -23,6 +23,9 @@ from prometheus_client import CollectorRegistry, Counter, Gauge, Info
from .types import StreamStatus
if TYPE_CHECKING:
from collections.abc import Callable, Generator
class NotificationType(StrEnum):
"""Notification type labels for the delivery counter."""
@@ -40,9 +43,6 @@ class ErrorSource(StrEnum):
COMMAND = "command"
if TYPE_CHECKING:
from collections.abc import Callable, Generator
# Mapping from StreamStatus enum to numeric gauge values
_STATUS_VALUES: dict[StreamStatus, float] = {
StreamStatus.ONLINE: 1.0,
+13 -11
View File
@@ -15,13 +15,16 @@
"""HTTP client for querying Owncast instance APIs."""
import json
from datetime import UTC, datetime
from http import HTTPStatus
from typing import TYPE_CHECKING, Any
import aiohttp
from .types import InvalidApiResponseError, StreamConfig, StreamState
from .types import (
InvalidApiResponseError,
StreamConfigObservation,
StreamStateObservation,
)
if TYPE_CHECKING:
import logging
@@ -110,14 +113,14 @@ class OwncastClient:
connector=connector,
)
async def get_stream_state(self, domain: str) -> StreamState | None:
async def get_stream_state(self, domain: str) -> StreamStateObservation | None:
"""Get the current stream state for a given domain.
HTTPS on port 443 is assumed, no other protocols or ports
are supported.
:param domain: The domain (not URL) where the stream is hosted.
:return: A StreamState if available, None on error.
:return: A StreamStateObservation if available, None on error.
"""
self.log.debug("[%s] Fetching current stream state...", domain)
with self.metrics.response_timer(domain) as timer:
@@ -126,10 +129,9 @@ class OwncastClient:
if new_state is None:
return None
observed_at = datetime.now(UTC)
try:
stream_state = StreamState.from_api_response(
new_state, domain, observed_at
stream_observation = StreamStateObservation.from_api_response(
new_state, domain
)
except InvalidApiResponseError as e:
self.log.warning(
@@ -142,16 +144,16 @@ class OwncastClient:
return None
timer.success()
return stream_state
return stream_observation
async def get_stream_config(self, domain: str) -> StreamConfig | None:
async def get_stream_config(self, domain: str) -> StreamConfigObservation | None:
"""Get the current stream config for a given domain.
HTTPS on port 443 is assumed, no other protocols or ports
are supported.
:param domain: The domain (not URL) where the stream is hosted.
:return: A StreamConfig, or None if fetch failed.
:return: A StreamConfigObservation, or None if fetch failed.
"""
self.log.debug("[%s] Fetching current stream config...", domain)
with self.metrics.response_timer(domain) as timer:
@@ -160,7 +162,7 @@ class OwncastClient:
return None
try:
stream_config = StreamConfig.from_api_response(config)
stream_config = StreamConfigObservation.from_api_response(config)
except InvalidApiResponseError as e:
self.log.warning(
"[%s] Rejecting response to request on %s as response "
+64 -20
View File
@@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Repository and schema upgrade definitions for OwncastSentry."""
"""Repository and schema upgrade definitions for OwncastSentry.
Only SQLite is supported as a database backend for now.
"""
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
@@ -35,9 +38,13 @@ if TYPE_CHECKING:
upgrade_table = UpgradeTable()
def _has_legacy_timestamp(value: Any) -> bool:
"""Return whether a legacy timestamp value carries usable content."""
return value is not None and str(value).strip() != ""
class _Unset:
"""Sentinel type for omitted partial stream update fields."""
__slots__ = ()
_UNSET = _Unset()
def _normalize_legacy_status_since(value: Any) -> str | None:
@@ -165,10 +172,9 @@ async def upgrade_v4(conn: Connection) -> None:
FROM streams"""
)
for row in rows:
online = _has_legacy_timestamp(row["last_connect_time"])
legacy_timestamp = (
row["last_connect_time"] if online else row["last_disconnect_time"]
)
last_connect_time = row["last_connect_time"]
online = last_connect_time is not None and str(last_connect_time).strip() != ""
legacy_timestamp = last_connect_time if online else row["last_disconnect_time"]
await conn.execute(
"""INSERT INTO streams_new (
domain, name, title, online, status_since, failure_counter
@@ -234,25 +240,63 @@ class StreamRepository:
result = await self.get_by_domain(domain)
return result is not None
async def update(self, state: StreamState) -> None:
"""Update a stream's state in the database.
async def update(
self,
domain: str,
*,
name: str | None | _Unset = _UNSET,
title: str | None | _Unset = _UNSET,
online: bool | _Unset = _UNSET,
status_since: str | None | _Unset = _UNSET,
) -> None:
"""Update only the supplied stream fields in the database.
This updates display/state fields only. Failure counters are
updated through dedicated methods.
Passing None writes NULL for nullable fields. Omitting a field leaves
that column unchanged.
:param state: The StreamState to save.
:param domain: The stream domain.
:param name: Optional stream display name update.
:param title: Optional stream title update.
:param online: Optional stream online state update.
:param status_since: Optional current status timestamp update.
"""
# _UNSET marks omitted fields so None can still be written as SQL NULL.
update_name = not isinstance(name, _Unset)
update_title = not isinstance(title, _Unset)
update_online = not isinstance(online, _Unset)
update_status_since = not isinstance(status_since, _Unset)
if not (update_name or update_title or update_online or update_status_since):
return
name_value = None if isinstance(name, _Unset) else name
title_value = None if isinstance(title, _Unset) else title
online_value = None if isinstance(online, _Unset) else online
status_since_value = None if isinstance(status_since, _Unset) else status_since
# SQLite's IS NOT gives null-safe comparisons for the supported backend.
query = """UPDATE streams
SET name=$1, title=$2, online=$3, status_since=$4
WHERE domain=$5"""
SET name = CASE WHEN $1 THEN $2 ELSE name END,
title = CASE WHEN $3 THEN $4 ELSE title END,
online = CASE WHEN $5 THEN $6 ELSE online END,
status_since = CASE WHEN $7 THEN $8 ELSE status_since END
WHERE domain=$9
AND (($1 AND name IS NOT $2)
OR ($3 AND title IS NOT $4)
OR ($5 AND online IS NOT $6)
OR ($7 AND status_since IS NOT $8))"""
async with self.db.acquire() as conn:
await conn.execute(
query,
state.name,
state.title,
state.online,
state.status_since,
state.domain,
update_name,
name_value,
update_title,
title_value,
update_online,
online_value,
update_status_since,
status_since_value,
domain,
)
async def delete(self, domain: str) -> None:
+194 -130
View File
@@ -16,9 +16,18 @@
import asyncio
import time
from dataclasses import dataclass
from datetime import UTC, datetime
from enum import Enum, auto
from typing import TYPE_CHECKING
from .types import StreamState, UpdateResult
from .types import (
StreamConfigObservation,
StreamState,
StreamStateObservation,
StreamStatus,
UpdateResult,
)
if TYPE_CHECKING:
import logging
@@ -56,6 +65,102 @@ def _should_query_stream(failure_counter: int) -> bool:
return failure_counter % _QUERY_EVERY_15_MINUTES_INTERVAL == 0
class _StreamTransitionKind(Enum):
"""High-level status transition from stored state to fresh observation."""
FIRST_OBSERVATION = auto()
WENT_LIVE = auto()
WENT_OFFLINE = auto()
TITLE_CHANGED = auto()
STATUS_UNCHANGED = auto()
@dataclass(frozen=True, slots=True)
class _StreamTransition:
"""Classified stream transition and timestamp metadata."""
kind: _StreamTransitionKind
status_since: str | None
class _NotificationKind(Enum):
"""Live/title notification action for a transition."""
NONE = auto()
LIVE = auto()
TITLE_CHANGE = auto()
@property
def should_notify(self) -> bool:
"""Return whether a live/title-change notification should be attempted."""
return self is not _NotificationKind.NONE
def _classify_transition(
old_state: StreamState, observation: StreamStateObservation
) -> _StreamTransition:
"""Classify the transition from the stored state to the latest observation."""
if old_state.status_since is None:
return _StreamTransition(
kind=_StreamTransitionKind.FIRST_OBSERVATION,
status_since=observation.observed_at,
)
if old_state.online != observation.online:
return _StreamTransition(
kind=(
_StreamTransitionKind.WENT_LIVE
if observation.online
else _StreamTransitionKind.WENT_OFFLINE
),
status_since=observation.observed_at,
)
if old_state.online and old_state.title != observation.title:
return _StreamTransition(
kind=_StreamTransitionKind.TITLE_CHANGED,
status_since=old_state.status_since,
)
return _StreamTransition(
kind=_StreamTransitionKind.STATUS_UNCHANGED,
status_since=old_state.status_since,
)
def _classify_notification(
transition: _StreamTransition,
old_state: StreamState,
observation: StreamStateObservation,
*,
offline_duration_seconds: int | None = None,
went_offline_after_last_notification: bool = False,
) -> _NotificationKind:
"""Classify the live/title notification policy for a transition.
The caller supplies timing context because it comes from the monitor's
monotonic offline cache and notification-service cache, not the transition.
"""
match transition.kind:
case _StreamTransitionKind.WENT_LIVE:
if offline_duration_seconds is not None and (
offline_duration_seconds < _TEMPORARY_OFFLINE_NOTIFICATION_COOLDOWN
):
if old_state.title != observation.title:
return _NotificationKind.TITLE_CHANGE
return _NotificationKind.NONE
return _NotificationKind.LIVE
case _StreamTransitionKind.TITLE_CHANGED:
if went_offline_after_last_notification:
return _NotificationKind.LIVE
return _NotificationKind.TITLE_CHANGE
case _:
return _NotificationKind.NONE
class StreamMonitor:
"""Monitors Owncast streams and detects state changes."""
@@ -178,22 +283,11 @@ class StreamMonitor:
# Backoff is expected behavior, not a failure
return True
# Flag: no status timestamp has been recorded yet, so suppress
# notifications for a stream's first observed state.
first_update = False
# Flag: whether to update the stream's state in the database.
# Used to avoid writes when state hasn't changed at all.
update_database = False
# The stream's latest configuration, if fetched during update.
stream_config = None
# Fetch the latest stream state from the server
new_state = await self.owncast_client.get_stream_state(domain)
observation = await self.owncast_client.get_stream_state(domain)
# If the fetch failed, increment failure counter and skip the update
if new_state is None:
if observation is None:
await self.stream_repo.increment_failure_counter(domain)
self.log.warning(
"[%s] Connection failure (counter=%s)",
@@ -216,138 +310,108 @@ class StreamMonitor:
# Initialize timer cache entries to prevent KeyError on first access
self.offline_timer_cache.setdefault(domain, 0)
if old_state.status_since is None:
# No stream history has been recorded yet. Don't send notifications.
update_database = True
first_update = True
transition = _classify_transition(old_state, observation)
# Notification policy may need runtime timing context in addition to
# the stored state and latest observation.
offline_duration_seconds: int | None = None
went_offline_after_last_notification = False
if first_update:
self.log.info(
"[%s] Not sending notifications. This is the first state "
"update for this stream.",
domain,
)
# Did the stream become publicly online?
elif new_state.online and not old_state.online:
# Yes! This stream is now live.
update_database = True
stream_config = await self.owncast_client.get_stream_config(domain)
self.log.info("[%s] Stream is now live!", domain)
# Calculate seconds since the stream last went offline
seconds_since_last_offline = round(
time.monotonic() - self.offline_timer_cache[domain]
)
# Use fallback values if config fetch failed
stream_name = stream_config.name if stream_config else domain
stream_tags = stream_config.tags if stream_config else ()
# Has this stream been offline for a short time?
if seconds_since_last_offline < _TEMPORARY_OFFLINE_NOTIFICATION_COOLDOWN:
# Did the stream title change?
if old_state.title != new_state.title:
# Stream was briefly down; send title change notification.
await self.notification_service.notify_stream_live(
domain,
stream_name,
new_state.title or "",
stream_tags,
title_change=True,
)
else:
# Briefly offline, no title change. Skip.
match transition.kind:
case _StreamTransitionKind.FIRST_OBSERVATION:
self.log.info(
"[%s] Not sending notifications. This is the first state "
"update for this stream.",
domain,
)
case _StreamTransitionKind.WENT_LIVE:
self.log.info("[%s] Stream is now live!", domain)
offline_duration_seconds = round(
time.monotonic() - self.offline_timer_cache[domain]
)
# Brief reconnects with the same title are treated as transient
# outages, not new live events.
if (
offline_duration_seconds < _TEMPORARY_OFFLINE_NOTIFICATION_COOLDOWN
and old_state.title == observation.title
):
self.log.info(
"[%s] Not sending notifications. Stream was only "
"offline for %s of %s seconds and did not change "
"its title.",
"[%s] Not sending notifications. Stream was only offline for "
"%s of %s seconds and did not change its title.",
domain,
seconds_since_last_offline,
offline_duration_seconds,
_TEMPORARY_OFFLINE_NOTIFICATION_COOLDOWN,
)
else:
# Offline for a while. Send a normal notification.
await self.notification_service.notify_stream_live(
domain,
stream_name,
new_state.title or "",
stream_tags,
title_change=False,
)
elif new_state.online and old_state.online:
# Did the stream title change mid-session?
if old_state.title != new_state.title:
case _StreamTransitionKind.TITLE_CHANGED:
self.log.info("[%s] Stream title was changed!", domain)
update_database = True
stream_config = await self.owncast_client.get_stream_config(domain)
# Use fallback values if config fetch failed
stream_name = stream_config.name if stream_config else domain
stream_tags = stream_config.tags if stream_config else ()
# If the offline marker is newer than the last recorded
# notification, send a regular go-live instead of a title
# change to avoid confusion.
if self.offline_timer_cache[
# If we saw an offline event after the last notification, send
# a go-live notice instead of only a title-change notice.
went_offline_after_last_notification = self.offline_timer_cache[
domain
] > self.notification_service.get_last_notification_time(domain):
await self.notification_service.notify_stream_live(
domain,
stream_name,
new_state.title or "",
stream_tags,
title_change=False,
)
else:
# No. Send a normal title change notification.
await self.notification_service.notify_stream_live(
domain,
stream_name,
new_state.title or "",
stream_tags,
title_change=True,
)
] > self.notification_service.get_last_notification_time(domain)
case _StreamTransitionKind.WENT_OFFLINE:
self.offline_timer_cache[domain] = time.monotonic()
self.log.info("[%s] Stream is now offline.", domain)
case _StreamTransitionKind.STATUS_UNCHANGED:
pass
# Did the stream go offline?
elif not new_state.online and old_state.online:
# Yep. This stream is now offline. Log it.
update_database = True
self.offline_timer_cache[domain] = time.monotonic()
self.log.info("[%s] Stream is now offline.", domain)
notification_kind = _classify_notification(
transition,
old_state,
observation,
offline_duration_seconds=offline_duration_seconds,
went_offline_after_last_notification=went_offline_after_last_notification,
)
# Update the database with current stream state, if needed.
if update_database:
# Ensure we have the stream config before updating
if stream_config is None:
stream_config = await self.owncast_client.get_stream_config(domain)
stream_config: StreamConfigObservation | None = None
observed_at = datetime.fromisoformat(observation.observed_at)
hourly_config_refresh_due = (
observed_at.tzinfo is not None and observed_at.astimezone(UTC).minute == 0
)
should_fetch_config = (
notification_kind.should_notify
or transition.kind is _StreamTransitionKind.FIRST_OBSERVATION
or hourly_config_refresh_due
)
if should_fetch_config:
stream_config = await self.owncast_client.get_stream_config(domain)
# Use fallback value if config fetch failed
stream_name = stream_config.name if stream_config else ""
if notification_kind.should_notify:
stream_name = stream_config.name if stream_config else domain
stream_tags = stream_config.tags if stream_config else ()
is_title_change = notification_kind is _NotificationKind.TITLE_CHANGE
self.log.debug("[%s] Updating stream state in database...", domain)
if first_update or old_state.online != new_state.online:
status_since = new_state.status_since
else:
status_since = old_state.status_since
# Create updated state object (title already truncated in new_state)
updated_state = StreamState(
domain=domain,
name=stream_name,
title=new_state.title,
online=new_state.online,
status_since=status_since,
await self.notification_service.notify_stream_live(
domain,
stream_name,
observation.title or "",
stream_tags,
title_change=is_title_change,
)
await self.stream_repo.update(updated_state)
self.log.debug("[%s] Saving stream state if changed...", domain)
if stream_config is not None:
await self.stream_repo.update(
domain,
name=stream_config.name,
title=observation.title,
online=observation.online,
status_since=transition.status_since,
)
else:
await self.stream_repo.update(
domain,
title=observation.title,
online=observation.online,
status_since=transition.status_since,
)
# All done.
self.log.debug("[%s] State update completed.", domain)
self.metrics.set_stream_status(domain, new_state.status)
stream_status = (
StreamStatus.ONLINE if observation.online else StreamStatus.OFFLINE
)
self.metrics.set_stream_status(domain, stream_status)
return True
async def _check_cleanup_thresholds(self, domain: str, counter: int) -> None:
+60 -31
View File
@@ -14,7 +14,7 @@
"""Data containers and domain errors for OwncastSentry."""
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import UTC, datetime
from enum import Enum
from typing import Any
@@ -80,6 +80,11 @@ def format_status_since(timestamp: datetime) -> str:
return timestamp.astimezone(UTC).isoformat(timespec="seconds")
def _current_observed_at() -> str:
"""Return the current UTC time in the package's timestamp format."""
return format_status_since(datetime.now(UTC))
class StreamStatus(Enum):
"""Represents the status of a stream."""
@@ -88,9 +93,51 @@ class StreamStatus(Enum):
UNKNOWN = "unknown"
@dataclass(frozen=True, slots=True)
class StreamStateObservation:
"""Status API sample fetched from an Owncast instance.
This records what `/api/status` reported during one request. `observed_at`
belongs to that fetch; the monitor decides whether that time starts a new
persisted online/offline period in `StreamState`.
"""
domain: str
title: str | None = None
online: bool = False
observed_at: str = field(default_factory=_current_observed_at)
@classmethod
def from_api_response(
cls, response: dict[str, Any], domain: str
) -> StreamStateObservation:
"""Create a StreamStateObservation from a status API response.
:param response: API response as a dictionary (camelCase keys).
:param domain: The stream domain.
:return: StreamStateObservation instance.
:raises InvalidApiResponseError: If the response shape is invalid.
"""
stream_title = _require_str(response, "streamTitle")
online = _require_field(response, "online")
if not isinstance(online, bool):
raise InvalidApiResponseError("online must be a boolean")
return cls(
domain=domain,
title=_truncate(stream_title, _MAX_STREAM_TITLE_LENGTH),
online=online,
)
@dataclass(frozen=True, slots=True)
class StreamState:
"""Represents the state of an Owncast stream."""
"""Stream record remembered by the repository.
States include persisted display data, the current online/offline period,
and failure counters used to derive user-facing status. `status_since`
records when the persisted online/offline period began.
"""
domain: str
name: str | None = None
@@ -112,30 +159,6 @@ class StreamState:
return StreamStatus.ONLINE
return StreamStatus.OFFLINE
@classmethod
def from_api_response(
cls, response: dict[str, Any], domain: str, observed_at: datetime
) -> StreamState:
"""Create a StreamState from an API response.
:param response: API response as a dictionary (camelCase keys).
:param domain: The stream domain.
:param observed_at: Local time when this status was observed.
:return: StreamState instance.
:raises InvalidApiResponseError: If the response shape is invalid.
"""
stream_title = _require_str(response, "streamTitle")
online = _require_field(response, "online")
if not isinstance(online, bool):
raise InvalidApiResponseError("online must be a boolean")
return cls(
domain=domain,
title=_truncate(stream_title, _MAX_STREAM_TITLE_LENGTH),
online=online,
status_since=format_status_since(observed_at),
)
@classmethod
def from_db_row(cls, row: dict[str, Any]) -> StreamState:
"""Create a StreamState from a database row.
@@ -154,18 +177,24 @@ class StreamState:
@dataclass(frozen=True, slots=True)
class StreamConfig:
"""Represents the configuration of an Owncast stream."""
class StreamConfigObservation:
"""Config API sample fetched from an Owncast instance.
This records display metadata from one `/api/config` request. It is used for
notification text and may refresh persisted display fields independently of
the stream's online/offline state.
"""
name: str = ""
tags: tuple[str, ...] = ()
observed_at: str = field(default_factory=_current_observed_at)
@classmethod
def from_api_response(cls, response: dict[str, Any]) -> StreamConfig:
"""Create a StreamConfig from an API response.
def from_api_response(cls, response: dict[str, Any]) -> StreamConfigObservation:
"""Create a StreamConfigObservation from an API response.
:param response: API response as a dictionary.
:return: StreamConfig instance.
:return: StreamConfigObservation instance.
:raises InvalidApiResponseError: If the response shape is invalid.
"""
# Apply Owncast's instance name limit.