this repo has no description
Implement complete thicket CLI application
Core Features:
- Modern CLI with Typer + Rich for beautiful terminal output
- Universal feed parser supporting RSS/Atom with auto-discovery
- Git storage system with structured JSON entries
- Duplicate management with manual curation
- Comprehensive test suite with pytest
Components:
- Data models: Pydantic models for config, feeds, users
- Core logic: FeedParser and GitStore classes
- CLI commands: init, add, sync, list, duplicates
- Tests: Complete test coverage for all components
- Documentation: README with usage examples
Architecture:
- src/thicket/ package structure
- Async HTTP with httpx for feed fetching
- HTML sanitization with bleach for security
- Modern Python packaging with pyproject.toml
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
+2908
+178
README.md
+178
README.md
···
1
+
# Thicket
2
+
3
+
A modern CLI tool for persisting Atom/RSS feeds in Git repositories, designed to enable distributed webblog comment structures.
4
+
5
+
## Features
6
+
7
+
- **Feed Auto-Discovery**: Automatically extracts user metadata from Atom/RSS feeds
8
+
- **Git Storage**: Stores feed entries in a Git repository with full history
9
+
- **Duplicate Management**: Manual curation of duplicate entries across feeds
10
+
- **Modern CLI**: Built with Typer and Rich for beautiful terminal output
11
+
- **Comprehensive Parsing**: Supports RSS 0.9x, RSS 1.0, RSS 2.0, and Atom feeds
12
+
- **Cron-Friendly**: Designed for scheduled execution
13
+
14
+
## Installation
15
+
16
+
```bash
17
+
# Install from source
18
+
pip install -e .
19
+
20
+
# Or install with dev dependencies
21
+
pip install -e .[dev]
22
+
```
23
+
24
+
## Quick Start
25
+
26
+
1. **Initialize a new thicket repository:**
27
+
```bash
28
+
thicket init ./my-feeds
29
+
```
30
+
31
+
2. **Add a user with their feed:**
32
+
```bash
33
+
thicket add user "alice" --feed "https://alice.example.com/feed.xml"
34
+
```
35
+
36
+
3. **Sync feeds to download entries:**
37
+
```bash
38
+
thicket sync --all
39
+
```
40
+
41
+
4. **List users and feeds:**
42
+
```bash
43
+
thicket list users
44
+
thicket list feeds
45
+
thicket list entries
46
+
```
47
+
48
+
## Commands
49
+
50
+
### Initialize
51
+
```bash
52
+
thicket init <git-store-path> [--cache-dir <path>] [--config <config-file>]
53
+
```
54
+
55
+
### Add Users and Feeds
56
+
```bash
57
+
# Add user with auto-discovery
58
+
thicket add user "username" --feed "https://example.com/feed.xml"
59
+
60
+
# Add user with manual metadata
61
+
thicket add user "username" \
62
+
--feed "https://example.com/feed.xml" \
63
+
--email "user@example.com" \
64
+
--homepage "https://example.com" \
65
+
--display-name "User Name"
66
+
67
+
# Add additional feed to existing user
68
+
thicket add feed "username" "https://example.com/other-feed.xml"
69
+
```
70
+
71
+
### Sync Feeds
72
+
```bash
73
+
# Sync all users
74
+
thicket sync --all
75
+
76
+
# Sync specific user
77
+
thicket sync --user "username"
78
+
79
+
# Dry run (preview changes)
80
+
thicket sync --all --dry-run
81
+
```
82
+
83
+
### List Information
84
+
```bash
85
+
# List all users
86
+
thicket list users
87
+
88
+
# List all feeds
89
+
thicket list feeds
90
+
91
+
# List feeds for specific user
92
+
thicket list feeds --user "username"
93
+
94
+
# List recent entries
95
+
thicket list entries --limit 20
96
+
97
+
# List entries for specific user
98
+
thicket list entries --user "username"
99
+
```
100
+
101
+
### Manage Duplicates
102
+
```bash
103
+
# List duplicate mappings
104
+
thicket duplicates list
105
+
106
+
# Mark entries as duplicates
107
+
thicket duplicates add "https://example.com/dup" "https://example.com/canonical"
108
+
109
+
# Remove duplicate mapping
110
+
thicket duplicates remove "https://example.com/dup"
111
+
```
112
+
113
+
## Configuration
114
+
115
+
Thicket uses a YAML configuration file (default: `thicket.yaml`):
116
+
117
+
```yaml
118
+
git_store: ./feeds-repo
119
+
cache_dir: ~/.cache/thicket
120
+
users:
121
+
- username: alice
122
+
feeds:
123
+
- https://alice.example.com/feed.xml
124
+
email: alice@example.com
125
+
homepage: https://alice.example.com
126
+
display_name: Alice
127
+
```
128
+
129
+
## Git Repository Structure
130
+
131
+
```
132
+
feeds-repo/
133
+
├── index.json # User directory index
134
+
├── duplicates.json # Duplicate entry mappings
135
+
├── alice/
136
+
│ ├── metadata.json # User metadata
137
+
│ ├── entry_id_1.json # Feed entries
138
+
│ └── entry_id_2.json
139
+
└── bob/
140
+
└── ...
141
+
```
142
+
143
+
## Development
144
+
145
+
### Setup
146
+
```bash
147
+
# Install in development mode
148
+
pip install -e .[dev]
149
+
150
+
# Run tests
151
+
pytest
152
+
153
+
# Run linting
154
+
ruff check src/
155
+
black --check src/
156
+
157
+
# Run type checking
158
+
mypy src/
159
+
```
160
+
161
+
### Architecture
162
+
163
+
- **CLI**: Modern interface with Typer and Rich
164
+
- **Feed Processing**: Universal parsing with feedparser
165
+
- **Git Storage**: Structured storage with GitPython
166
+
- **Data Models**: Pydantic for validation and serialization
167
+
- **Async HTTP**: httpx for efficient feed fetching
168
+
169
+
## Use Cases
170
+
171
+
- **Blog Aggregation**: Collect and archive blog posts from multiple sources
172
+
- **Comment Networks**: Enable distributed commenting systems
173
+
- **Feed Archival**: Preserve feed history beyond typical feed depth limits
174
+
- **Content Curation**: Manage and deduplicate content across feeds
175
+
176
+
## License
177
+
178
+
MIT License - see LICENSE file for details.
+169
pyproject.toml
+169
pyproject.toml
···
1
+
[build-system]
2
+
requires = ["hatchling"]
3
+
build-backend = "hatchling.build"
4
+
5
+
[project]
6
+
name = "thicket"
7
+
dynamic = ["version"]
8
+
description = "A CLI tool for persisting Atom/RSS feeds in Git repositories"
9
+
readme = "README.md"
10
+
license = "MIT"
11
+
requires-python = ">=3.9"
12
+
authors = [
13
+
{name = "thicket", email = "thicket@example.com"},
14
+
]
15
+
classifiers = [
16
+
"Development Status :: 3 - Alpha",
17
+
"Intended Audience :: Developers",
18
+
"License :: OSI Approved :: MIT License",
19
+
"Operating System :: OS Independent",
20
+
"Programming Language :: Python :: 3",
21
+
"Programming Language :: Python :: 3.9",
22
+
"Programming Language :: Python :: 3.10",
23
+
"Programming Language :: Python :: 3.11",
24
+
"Programming Language :: Python :: 3.12",
25
+
"Programming Language :: Python :: 3.13",
26
+
"Topic :: Internet :: WWW/HTTP :: Dynamic Content :: News/Diary",
27
+
"Topic :: Software Development :: Version Control :: Git",
28
+
"Topic :: Text Processing :: Markup :: XML",
29
+
]
30
+
dependencies = [
31
+
"typer>=0.15.0",
32
+
"rich>=13.0.0",
33
+
"GitPython>=3.1.40",
34
+
"feedparser>=6.0.11",
35
+
"pydantic>=2.11.0",
36
+
"pydantic-settings>=2.10.0",
37
+
"httpx>=0.28.0",
38
+
"pendulum>=3.0.0",
39
+
"bleach>=6.0.0",
40
+
"platformdirs>=4.0.0",
41
+
"pyyaml>=6.0.0",
42
+
]
43
+
44
+
[project.optional-dependencies]
45
+
dev = [
46
+
"pytest>=8.0.0",
47
+
"pytest-asyncio>=0.24.0",
48
+
"pytest-cov>=6.0.0",
49
+
"black>=24.0.0",
50
+
"ruff>=0.8.0",
51
+
"mypy>=1.13.0",
52
+
"types-PyYAML>=6.0.0",
53
+
]
54
+
55
+
[project.urls]
56
+
Homepage = "https://github.com/example/thicket"
57
+
Documentation = "https://github.com/example/thicket"
58
+
Repository = "https://github.com/example/thicket"
59
+
"Bug Tracker" = "https://github.com/example/thicket/issues"
60
+
61
+
[project.scripts]
62
+
thicket = "thicket.cli.main:app"
63
+
64
+
[tool.hatch.version]
65
+
path = "src/thicket/__init__.py"
66
+
67
+
[tool.hatch.build.targets.wheel]
68
+
packages = ["src/thicket"]
69
+
70
+
[tool.black]
71
+
line-length = 88
72
+
target-version = ['py39']
73
+
include = '\.pyi?$'
74
+
extend-exclude = '''
75
+
/(
76
+
# directories
77
+
\.eggs
78
+
| \.git
79
+
| \.hg
80
+
| \.mypy_cache
81
+
| \.tox
82
+
| \.venv
83
+
| build
84
+
| dist
85
+
)/
86
+
'''
87
+
88
+
[tool.ruff]
89
+
target-version = "py39"
90
+
line-length = 88
91
+
select = [
92
+
"E", # pycodestyle errors
93
+
"W", # pycodestyle warnings
94
+
"F", # pyflakes
95
+
"I", # isort
96
+
"B", # flake8-bugbear
97
+
"C4", # flake8-comprehensions
98
+
"UP", # pyupgrade
99
+
]
100
+
ignore = [
101
+
"E501", # line too long, handled by black
102
+
"B008", # do not perform function calls in argument defaults
103
+
"C901", # too complex
104
+
]
105
+
106
+
[tool.ruff.per-file-ignores]
107
+
"__init__.py" = ["F401"]
108
+
109
+
[tool.mypy]
110
+
python_version = "3.9"
111
+
check_untyped_defs = true
112
+
disallow_any_generics = true
113
+
disallow_incomplete_defs = true
114
+
disallow_untyped_defs = true
115
+
no_implicit_optional = true
116
+
warn_redundant_casts = true
117
+
warn_unused_ignores = true
118
+
warn_return_any = true
119
+
strict_optional = true
120
+
121
+
[[tool.mypy.overrides]]
122
+
module = [
123
+
"feedparser",
124
+
"git",
125
+
"bleach",
126
+
]
127
+
ignore_missing_imports = true
128
+
129
+
[tool.pytest.ini_options]
130
+
testpaths = ["tests"]
131
+
python_files = ["test_*.py"]
132
+
python_classes = ["Test*"]
133
+
python_functions = ["test_*"]
134
+
addopts = [
135
+
"-ra",
136
+
"--strict-markers",
137
+
"--strict-config",
138
+
"--cov=src/thicket",
139
+
"--cov-report=term-missing",
140
+
"--cov-report=html",
141
+
"--cov-report=xml",
142
+
]
143
+
filterwarnings = [
144
+
"error",
145
+
"ignore::UserWarning",
146
+
"ignore::DeprecationWarning",
147
+
]
148
+
markers = [
149
+
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
150
+
"integration: marks tests as integration tests",
151
+
]
152
+
153
+
[tool.coverage.run]
154
+
source = ["src"]
155
+
branch = true
156
+
157
+
[tool.coverage.report]
158
+
exclude_lines = [
159
+
"pragma: no cover",
160
+
"def __repr__",
161
+
"if self.debug:",
162
+
"if settings.DEBUG",
163
+
"raise AssertionError",
164
+
"raise NotImplementedError",
165
+
"if 0:",
166
+
"if __name__ == .__main__.:",
167
+
"class .*\\bProtocol\\):",
168
+
"@(abc\\.)?abstractmethod",
169
+
]
+5
src/thicket/__init__.py
+5
src/thicket/__init__.py
+6
src/thicket/__main__.py
+6
src/thicket/__main__.py
+5
src/thicket/cli/__init__.py
+5
src/thicket/cli/__init__.py
+6
src/thicket/cli/commands/__init__.py
+6
src/thicket/cli/commands/__init__.py
+193
src/thicket/cli/commands/add.py
+193
src/thicket/cli/commands/add.py
···
1
+
"""Add command for thicket."""
2
+
3
+
import asyncio
4
+
from pathlib import Path
5
+
from typing import Optional
6
+
7
+
import typer
8
+
from pydantic import HttpUrl, ValidationError
9
+
10
+
from ...core.feed_parser import FeedParser
11
+
from ...core.git_store import GitStore
12
+
from ...models import UserConfig
13
+
from ..main import app
14
+
from ..utils import (
15
+
create_progress,
16
+
load_config,
17
+
print_error,
18
+
print_info,
19
+
print_success,
20
+
save_config,
21
+
)
22
+
23
+
24
+
@app.command("add")
25
+
def add_command(
26
+
subcommand: str = typer.Argument(..., help="Subcommand: 'user' or 'feed'"),
27
+
username: str = typer.Argument(..., help="Username"),
28
+
feed_url: Optional[str] = typer.Argument(None, help="Feed URL (required for 'user' command)"),
29
+
email: Optional[str] = typer.Option(None, "--email", "-e", help="User email"),
30
+
homepage: Optional[str] = typer.Option(None, "--homepage", "-h", help="User homepage"),
31
+
icon: Optional[str] = typer.Option(None, "--icon", "-i", help="User icon URL"),
32
+
display_name: Optional[str] = typer.Option(None, "--display-name", "-d", help="User display name"),
33
+
config_file: Optional[Path] = typer.Option(
34
+
Path("thicket.yaml"), "--config", help="Configuration file path"
35
+
),
36
+
auto_discover: bool = typer.Option(
37
+
True, "--auto-discover/--no-auto-discover", help="Auto-discover user metadata from feed"
38
+
),
39
+
) -> None:
40
+
"""Add a user or feed to thicket."""
41
+
42
+
if subcommand == "user":
43
+
add_user(username, feed_url, email, homepage, icon, display_name, config_file, auto_discover)
44
+
elif subcommand == "feed":
45
+
add_feed(username, feed_url, config_file)
46
+
else:
47
+
print_error(f"Unknown subcommand: {subcommand}")
48
+
print_error("Use 'user' or 'feed'")
49
+
raise typer.Exit(1)
50
+
51
+
52
+
def add_user(
53
+
username: str,
54
+
feed_url: Optional[str],
55
+
email: Optional[str],
56
+
homepage: Optional[str],
57
+
icon: Optional[str],
58
+
display_name: Optional[str],
59
+
config_file: Path,
60
+
auto_discover: bool,
61
+
) -> None:
62
+
"""Add a new user with feed."""
63
+
64
+
if not feed_url:
65
+
print_error("Feed URL is required when adding a user")
66
+
raise typer.Exit(1)
67
+
68
+
# Validate feed URL
69
+
try:
70
+
validated_feed_url = HttpUrl(feed_url)
71
+
except ValidationError:
72
+
print_error(f"Invalid feed URL: {feed_url}")
73
+
raise typer.Exit(1)
74
+
75
+
# Load configuration
76
+
config = load_config(config_file)
77
+
78
+
# Check if user already exists
79
+
existing_user = config.find_user(username)
80
+
if existing_user:
81
+
print_error(f"User '{username}' already exists")
82
+
print_error("Use 'thicket add feed' to add additional feeds")
83
+
raise typer.Exit(1)
84
+
85
+
# Auto-discover metadata if enabled
86
+
discovered_metadata = None
87
+
if auto_discover:
88
+
discovered_metadata = asyncio.run(discover_feed_metadata(validated_feed_url))
89
+
90
+
# Create user config with manual overrides taking precedence
91
+
user_config = UserConfig(
92
+
username=username,
93
+
feeds=[validated_feed_url],
94
+
email=email or (discovered_metadata.author_email if discovered_metadata else None),
95
+
homepage=HttpUrl(homepage) if homepage else (discovered_metadata.author_uri or discovered_metadata.link if discovered_metadata else None),
96
+
icon=HttpUrl(icon) if icon else (discovered_metadata.logo or discovered_metadata.icon or discovered_metadata.image_url if discovered_metadata else None),
97
+
display_name=display_name or (discovered_metadata.author_name or discovered_metadata.title if discovered_metadata else None),
98
+
)
99
+
100
+
# Add user to configuration
101
+
config.add_user(user_config)
102
+
103
+
# Save configuration
104
+
save_config(config, config_file)
105
+
106
+
# Add user to Git store
107
+
git_store = GitStore(config.git_store)
108
+
git_store.add_user(
109
+
username=username,
110
+
display_name=user_config.display_name,
111
+
email=user_config.email,
112
+
homepage=str(user_config.homepage) if user_config.homepage else None,
113
+
icon=str(user_config.icon) if user_config.icon else None,
114
+
feeds=[str(f) for f in user_config.feeds],
115
+
)
116
+
117
+
# Commit changes
118
+
git_store.commit_changes(f"Add user: {username}")
119
+
120
+
print_success(f"Added user '{username}' with feed: {feed_url}")
121
+
122
+
if discovered_metadata and auto_discover:
123
+
print_info("Auto-discovered metadata:")
124
+
if user_config.display_name:
125
+
print_info(f" Display name: {user_config.display_name}")
126
+
if user_config.email:
127
+
print_info(f" Email: {user_config.email}")
128
+
if user_config.homepage:
129
+
print_info(f" Homepage: {user_config.homepage}")
130
+
if user_config.icon:
131
+
print_info(f" Icon: {user_config.icon}")
132
+
133
+
134
+
def add_feed(username: str, feed_url: Optional[str], config_file: Path) -> None:
135
+
"""Add a feed to an existing user."""
136
+
137
+
if not feed_url:
138
+
print_error("Feed URL is required")
139
+
raise typer.Exit(1)
140
+
141
+
# Validate feed URL
142
+
try:
143
+
validated_feed_url = HttpUrl(feed_url)
144
+
except ValidationError:
145
+
print_error(f"Invalid feed URL: {feed_url}")
146
+
raise typer.Exit(1)
147
+
148
+
# Load configuration
149
+
config = load_config(config_file)
150
+
151
+
# Check if user exists
152
+
user = config.find_user(username)
153
+
if not user:
154
+
print_error(f"User '{username}' not found")
155
+
print_error("Use 'thicket add user' to add a new user")
156
+
raise typer.Exit(1)
157
+
158
+
# Check if feed already exists
159
+
if validated_feed_url in user.feeds:
160
+
print_error(f"Feed already exists for user '{username}': {feed_url}")
161
+
raise typer.Exit(1)
162
+
163
+
# Add feed to user
164
+
if config.add_feed_to_user(username, validated_feed_url):
165
+
save_config(config, config_file)
166
+
167
+
# Update Git store
168
+
git_store = GitStore(config.git_store)
169
+
git_store.update_user(username, feeds=[str(f) for f in user.feeds])
170
+
git_store.commit_changes(f"Add feed to user {username}: {feed_url}")
171
+
172
+
print_success(f"Added feed to user '{username}': {feed_url}")
173
+
else:
174
+
print_error(f"Failed to add feed to user '{username}'")
175
+
raise typer.Exit(1)
176
+
177
+
178
+
async def discover_feed_metadata(feed_url: HttpUrl):
179
+
"""Discover metadata from a feed URL."""
180
+
try:
181
+
with create_progress() as progress:
182
+
task = progress.add_task("Discovering feed metadata...", total=None)
183
+
184
+
parser = FeedParser()
185
+
content = await parser.fetch_feed(feed_url)
186
+
metadata, _ = parser.parse_feed(content, feed_url)
187
+
188
+
progress.update(task, completed=True)
189
+
return metadata
190
+
191
+
except Exception as e:
192
+
print_error(f"Failed to discover feed metadata: {e}")
193
+
return None
+121
src/thicket/cli/commands/duplicates.py
+121
src/thicket/cli/commands/duplicates.py
···
1
+
"""Duplicates command for thicket."""
2
+
3
+
from pathlib import Path
4
+
from typing import Optional
5
+
6
+
import typer
7
+
from rich.table import Table
8
+
9
+
from ...core.git_store import GitStore
10
+
from ..main import app
11
+
from ..utils import (
12
+
console,
13
+
load_config,
14
+
print_error,
15
+
print_info,
16
+
print_success,
17
+
)
18
+
19
+
20
+
@app.command("duplicates")
21
+
def duplicates_command(
22
+
action: str = typer.Argument(..., help="Action: 'list', 'add', 'remove'"),
23
+
duplicate_id: Optional[str] = typer.Argument(None, help="Duplicate entry ID"),
24
+
canonical_id: Optional[str] = typer.Argument(None, help="Canonical entry ID"),
25
+
config_file: Optional[Path] = typer.Option(
26
+
Path("thicket.yaml"), "--config", help="Configuration file path"
27
+
),
28
+
) -> None:
29
+
"""Manage duplicate entry mappings."""
30
+
31
+
# Load configuration
32
+
config = load_config(config_file)
33
+
34
+
# Initialize Git store
35
+
git_store = GitStore(config.git_store)
36
+
37
+
if action == "list":
38
+
list_duplicates(git_store)
39
+
elif action == "add":
40
+
add_duplicate(git_store, duplicate_id, canonical_id)
41
+
elif action == "remove":
42
+
remove_duplicate(git_store, duplicate_id)
43
+
else:
44
+
print_error(f"Unknown action: {action}")
45
+
print_error("Use 'list', 'add', or 'remove'")
46
+
raise typer.Exit(1)
47
+
48
+
49
+
def list_duplicates(git_store: GitStore) -> None:
50
+
"""List all duplicate mappings."""
51
+
duplicates = git_store.get_duplicates()
52
+
53
+
if not duplicates.duplicates:
54
+
print_info("No duplicate mappings found")
55
+
return
56
+
57
+
table = Table(title="Duplicate Entry Mappings")
58
+
table.add_column("Duplicate ID", style="red")
59
+
table.add_column("Canonical ID", style="green")
60
+
61
+
for duplicate_id, canonical_id in duplicates.duplicates.items():
62
+
table.add_row(duplicate_id, canonical_id)
63
+
64
+
console.print(table)
65
+
print_info(f"Total duplicates: {len(duplicates.duplicates)}")
66
+
67
+
68
+
def add_duplicate(git_store: GitStore, duplicate_id: Optional[str], canonical_id: Optional[str]) -> None:
69
+
"""Add a duplicate mapping."""
70
+
if not duplicate_id:
71
+
print_error("Duplicate ID is required")
72
+
raise typer.Exit(1)
73
+
74
+
if not canonical_id:
75
+
print_error("Canonical ID is required")
76
+
raise typer.Exit(1)
77
+
78
+
# Check if duplicate_id already exists
79
+
duplicates = git_store.get_duplicates()
80
+
if duplicates.is_duplicate(duplicate_id):
81
+
existing_canonical = duplicates.get_canonical(duplicate_id)
82
+
print_error(f"Duplicate ID already mapped to: {existing_canonical}")
83
+
print_error("Use 'remove' first to change the mapping")
84
+
raise typer.Exit(1)
85
+
86
+
# Check if we're trying to make a canonical ID point to itself
87
+
if duplicate_id == canonical_id:
88
+
print_error("Duplicate ID cannot be the same as canonical ID")
89
+
raise typer.Exit(1)
90
+
91
+
# Add the mapping
92
+
git_store.add_duplicate(duplicate_id, canonical_id)
93
+
94
+
# Commit changes
95
+
git_store.commit_changes(f"Add duplicate mapping: {duplicate_id} -> {canonical_id}")
96
+
97
+
print_success(f"Added duplicate mapping: {duplicate_id} -> {canonical_id}")
98
+
99
+
100
+
def remove_duplicate(git_store: GitStore, duplicate_id: Optional[str]) -> None:
101
+
"""Remove a duplicate mapping."""
102
+
if not duplicate_id:
103
+
print_error("Duplicate ID is required")
104
+
raise typer.Exit(1)
105
+
106
+
# Check if mapping exists
107
+
duplicates = git_store.get_duplicates()
108
+
if not duplicates.is_duplicate(duplicate_id):
109
+
print_error(f"No duplicate mapping found for: {duplicate_id}")
110
+
raise typer.Exit(1)
111
+
112
+
canonical_id = duplicates.get_canonical(duplicate_id)
113
+
114
+
# Remove the mapping
115
+
if git_store.remove_duplicate(duplicate_id):
116
+
# Commit changes
117
+
git_store.commit_changes(f"Remove duplicate mapping: {duplicate_id} -> {canonical_id}")
118
+
print_success(f"Removed duplicate mapping: {duplicate_id} -> {canonical_id}")
119
+
else:
120
+
print_error(f"Failed to remove duplicate mapping: {duplicate_id}")
121
+
raise typer.Exit(1)
+77
src/thicket/cli/commands/init.py
+77
src/thicket/cli/commands/init.py
···
1
+
"""Initialize command for thicket."""
2
+
3
+
from pathlib import Path
4
+
from typing import Optional
5
+
6
+
import typer
7
+
from pydantic import ValidationError
8
+
9
+
from ...core.git_store import GitStore
10
+
from ...models import ThicketConfig
11
+
from ..main import app
12
+
from ..utils import print_error, print_success, save_config
13
+
14
+
15
+
@app.command()
16
+
def init(
17
+
git_store: Path = typer.Argument(..., help="Path to Git repository for storing feeds"),
18
+
cache_dir: Optional[Path] = typer.Option(
19
+
None, "--cache-dir", "-c", help="Cache directory (default: ~/.cache/thicket)"
20
+
),
21
+
config_file: Optional[Path] = typer.Option(
22
+
None, "--config", help="Configuration file path (default: thicket.yaml)"
23
+
),
24
+
force: bool = typer.Option(
25
+
False, "--force", "-f", help="Overwrite existing configuration"
26
+
),
27
+
) -> None:
28
+
"""Initialize a new thicket configuration and Git store."""
29
+
30
+
# Set default paths
31
+
if cache_dir is None:
32
+
from platformdirs import user_cache_dir
33
+
cache_dir = Path(user_cache_dir("thicket"))
34
+
35
+
if config_file is None:
36
+
config_file = Path("thicket.yaml")
37
+
38
+
# Check if config already exists
39
+
if config_file.exists() and not force:
40
+
print_error(f"Configuration file already exists: {config_file}")
41
+
print_error("Use --force to overwrite")
42
+
raise typer.Exit(1)
43
+
44
+
# Create cache directory
45
+
cache_dir.mkdir(parents=True, exist_ok=True)
46
+
47
+
# Create Git store
48
+
try:
49
+
git_store_obj = GitStore(git_store)
50
+
print_success(f"Initialized Git store at: {git_store}")
51
+
except Exception as e:
52
+
print_error(f"Failed to initialize Git store: {e}")
53
+
raise typer.Exit(1)
54
+
55
+
# Create configuration
56
+
try:
57
+
config = ThicketConfig(
58
+
git_store=git_store,
59
+
cache_dir=cache_dir,
60
+
users=[]
61
+
)
62
+
63
+
save_config(config, config_file)
64
+
print_success(f"Created configuration file: {config_file}")
65
+
66
+
except ValidationError as e:
67
+
print_error(f"Invalid configuration: {e}")
68
+
raise typer.Exit(1)
69
+
except Exception as e:
70
+
print_error(f"Failed to create configuration: {e}")
71
+
raise typer.Exit(1)
72
+
73
+
print_success("Thicket initialized successfully!")
74
+
print_success(f"Git store: {git_store}")
75
+
print_success(f"Cache directory: {cache_dir}")
76
+
print_success(f"Configuration: {config_file}")
77
+
print_success("Run 'thicket add user' to add your first user and feed.")
+146
src/thicket/cli/commands/list_cmd.py
+146
src/thicket/cli/commands/list_cmd.py
···
1
+
"""List command for thicket."""
2
+
3
+
from pathlib import Path
4
+
from typing import Optional
5
+
6
+
import typer
7
+
from rich.table import Table
8
+
9
+
from ...core.git_store import GitStore
10
+
from ..main import app
11
+
from ..utils import (
12
+
console,
13
+
load_config,
14
+
print_error,
15
+
print_feeds_table,
16
+
print_info,
17
+
print_users_table,
18
+
)
19
+
20
+
21
+
@app.command("list")
22
+
def list_command(
23
+
what: str = typer.Argument(..., help="What to list: 'users', 'feeds', 'entries'"),
24
+
user: Optional[str] = typer.Option(
25
+
None, "--user", "-u", help="Filter by specific user"
26
+
),
27
+
limit: Optional[int] = typer.Option(
28
+
None, "--limit", "-l", help="Limit number of results"
29
+
),
30
+
config_file: Optional[Path] = typer.Option(
31
+
Path("thicket.yaml"), "--config", help="Configuration file path"
32
+
),
33
+
) -> None:
34
+
"""List users, feeds, or entries."""
35
+
36
+
# Load configuration
37
+
config = load_config(config_file)
38
+
39
+
if what == "users":
40
+
list_users(config)
41
+
elif what == "feeds":
42
+
list_feeds(config, user)
43
+
elif what == "entries":
44
+
list_entries(config, user, limit)
45
+
else:
46
+
print_error(f"Unknown list type: {what}")
47
+
print_error("Use 'users', 'feeds', or 'entries'")
48
+
raise typer.Exit(1)
49
+
50
+
51
+
def list_users(config) -> None:
52
+
"""List all users."""
53
+
if not config.users:
54
+
print_info("No users configured")
55
+
return
56
+
57
+
print_users_table(config)
58
+
59
+
60
+
def list_feeds(config, username: Optional[str] = None) -> None:
61
+
"""List feeds, optionally filtered by user."""
62
+
if username:
63
+
user = config.find_user(username)
64
+
if not user:
65
+
print_error(f"User '{username}' not found")
66
+
raise typer.Exit(1)
67
+
68
+
if not user.feeds:
69
+
print_info(f"No feeds configured for user '{username}'")
70
+
return
71
+
72
+
print_feeds_table(config, username)
73
+
74
+
75
+
def list_entries(config, username: Optional[str] = None, limit: Optional[int] = None) -> None:
76
+
"""List entries, optionally filtered by user."""
77
+
78
+
# Initialize Git store
79
+
git_store = GitStore(config.git_store)
80
+
81
+
if username:
82
+
# List entries for specific user
83
+
user = config.find_user(username)
84
+
if not user:
85
+
print_error(f"User '{username}' not found")
86
+
raise typer.Exit(1)
87
+
88
+
entries = git_store.list_entries(username, limit)
89
+
if not entries:
90
+
print_info(f"No entries found for user '{username}'")
91
+
return
92
+
93
+
print_entries_table([entries], [username])
94
+
95
+
else:
96
+
# List entries for all users
97
+
all_entries = []
98
+
all_usernames = []
99
+
100
+
for user in config.users:
101
+
entries = git_store.list_entries(user.username, limit)
102
+
if entries:
103
+
all_entries.append(entries)
104
+
all_usernames.append(user.username)
105
+
106
+
if not all_entries:
107
+
print_info("No entries found")
108
+
return
109
+
110
+
print_entries_table(all_entries, all_usernames)
111
+
112
+
113
+
def print_entries_table(entries_by_user: list[list], usernames: list[str]) -> None:
114
+
"""Print a table of entries."""
115
+
table = Table(title="Feed Entries")
116
+
table.add_column("User", style="cyan", no_wrap=True)
117
+
table.add_column("Title", style="bold")
118
+
table.add_column("Updated", style="blue")
119
+
table.add_column("URL", style="green")
120
+
121
+
# Combine all entries with usernames
122
+
all_entries = []
123
+
for entries, username in zip(entries_by_user, usernames):
124
+
for entry in entries:
125
+
all_entries.append((username, entry))
126
+
127
+
# Sort by updated time (newest first)
128
+
all_entries.sort(key=lambda x: x[1].updated, reverse=True)
129
+
130
+
for username, entry in all_entries:
131
+
# Format updated time
132
+
updated_str = entry.updated.strftime("%Y-%m-%d %H:%M")
133
+
134
+
# Truncate title if too long
135
+
title = entry.title
136
+
if len(title) > 50:
137
+
title = title[:47] + "..."
138
+
139
+
table.add_row(
140
+
username,
141
+
title,
142
+
updated_str,
143
+
str(entry.link),
144
+
)
145
+
146
+
console.print(table)
+142
src/thicket/cli/commands/sync.py
+142
src/thicket/cli/commands/sync.py
···
1
+
"""Sync command for thicket."""
2
+
3
+
import asyncio
4
+
from pathlib import Path
5
+
from typing import Optional
6
+
7
+
import typer
8
+
from rich.progress import track
9
+
10
+
from ...core.feed_parser import FeedParser
11
+
from ...core.git_store import GitStore
12
+
from ..main import app
13
+
from ..utils import (
14
+
create_progress,
15
+
load_config,
16
+
print_error,
17
+
print_info,
18
+
print_success,
19
+
)
20
+
21
+
22
+
@app.command()
23
+
def sync(
24
+
all_users: bool = typer.Option(
25
+
False, "--all", "-a", help="Sync all users and feeds"
26
+
),
27
+
user: Optional[str] = typer.Option(
28
+
None, "--user", "-u", help="Sync specific user only"
29
+
),
30
+
config_file: Optional[Path] = typer.Option(
31
+
Path("thicket.yaml"), "--config", help="Configuration file path"
32
+
),
33
+
dry_run: bool = typer.Option(
34
+
False, "--dry-run", help="Show what would be synced without making changes"
35
+
),
36
+
) -> None:
37
+
"""Sync feeds and store entries in Git repository."""
38
+
39
+
# Load configuration
40
+
config = load_config(config_file)
41
+
42
+
# Determine which users to sync
43
+
users_to_sync = []
44
+
if all_users:
45
+
users_to_sync = config.users
46
+
elif user:
47
+
user_config = config.find_user(user)
48
+
if not user_config:
49
+
print_error(f"User '{user}' not found")
50
+
raise typer.Exit(1)
51
+
users_to_sync = [user_config]
52
+
else:
53
+
print_error("Specify --all to sync all users or --user to sync a specific user")
54
+
raise typer.Exit(1)
55
+
56
+
if not users_to_sync:
57
+
print_info("No users configured to sync")
58
+
return
59
+
60
+
# Initialize Git store
61
+
git_store = GitStore(config.git_store)
62
+
63
+
# Sync each user
64
+
total_new_entries = 0
65
+
total_updated_entries = 0
66
+
67
+
for user_config in users_to_sync:
68
+
print_info(f"Syncing user: {user_config.username}")
69
+
70
+
user_new_entries = 0
71
+
user_updated_entries = 0
72
+
73
+
# Sync each feed for the user
74
+
for feed_url in track(user_config.feeds, description=f"Syncing {user_config.username}'s feeds"):
75
+
try:
76
+
new_entries, updated_entries = asyncio.run(
77
+
sync_feed(git_store, user_config.username, feed_url, dry_run)
78
+
)
79
+
user_new_entries += new_entries
80
+
user_updated_entries += updated_entries
81
+
82
+
except Exception as e:
83
+
print_error(f"Failed to sync feed {feed_url}: {e}")
84
+
continue
85
+
86
+
print_info(f"User {user_config.username}: {user_new_entries} new, {user_updated_entries} updated")
87
+
total_new_entries += user_new_entries
88
+
total_updated_entries += user_updated_entries
89
+
90
+
# Commit changes if not dry run
91
+
if not dry_run and (total_new_entries > 0 or total_updated_entries > 0):
92
+
commit_message = f"Sync feeds: {total_new_entries} new entries, {total_updated_entries} updated"
93
+
git_store.commit_changes(commit_message)
94
+
print_success(f"Committed changes: {commit_message}")
95
+
96
+
# Summary
97
+
if dry_run:
98
+
print_info(f"Dry run complete: would sync {total_new_entries} new entries, {total_updated_entries} updated")
99
+
else:
100
+
print_success(f"Sync complete: {total_new_entries} new entries, {total_updated_entries} updated")
101
+
102
+
103
+
async def sync_feed(git_store: GitStore, username: str, feed_url, dry_run: bool) -> tuple[int, int]:
104
+
"""Sync a single feed for a user."""
105
+
106
+
parser = FeedParser()
107
+
108
+
try:
109
+
# Fetch and parse feed
110
+
content = await parser.fetch_feed(feed_url)
111
+
metadata, entries = parser.parse_feed(content, feed_url)
112
+
113
+
new_entries = 0
114
+
updated_entries = 0
115
+
116
+
# Process each entry
117
+
for entry in entries:
118
+
try:
119
+
# Check if entry already exists
120
+
existing_entry = git_store.get_entry(username, entry.id)
121
+
122
+
if existing_entry:
123
+
# Check if entry has been updated
124
+
if existing_entry.updated != entry.updated:
125
+
if not dry_run:
126
+
git_store.store_entry(username, entry)
127
+
updated_entries += 1
128
+
else:
129
+
# New entry
130
+
if not dry_run:
131
+
git_store.store_entry(username, entry)
132
+
new_entries += 1
133
+
134
+
except Exception as e:
135
+
print_error(f"Failed to process entry {entry.id}: {e}")
136
+
continue
137
+
138
+
return new_entries, updated_entries
139
+
140
+
except Exception as e:
141
+
print_error(f"Failed to sync feed {feed_url}: {e}")
142
+
return 0, 0
+45
src/thicket/cli/main.py
+45
src/thicket/cli/main.py
···
1
+
"""Main CLI application using Typer."""
2
+
3
+
import typer
4
+
from rich.console import Console
5
+
6
+
from .. import __version__
7
+
8
+
app = typer.Typer(
9
+
name="thicket",
10
+
help="A CLI tool for persisting Atom/RSS feeds in Git repositories",
11
+
no_args_is_help=True,
12
+
rich_markup_mode="rich",
13
+
)
14
+
15
+
console = Console()
16
+
17
+
18
+
def version_callback(value: bool) -> None:
19
+
"""Show version and exit."""
20
+
if value:
21
+
console.print(f"thicket version {__version__}")
22
+
raise typer.Exit()
23
+
24
+
25
+
@app.callback()
26
+
def main(
27
+
version: bool = typer.Option(
28
+
None,
29
+
"--version",
30
+
"-v",
31
+
help="Show the version and exit",
32
+
callback=version_callback,
33
+
is_eager=True,
34
+
),
35
+
) -> None:
36
+
"""Thicket: A CLI tool for persisting Atom/RSS feeds in Git repositories."""
37
+
pass
38
+
39
+
40
+
# Import commands to register them
41
+
from .commands import duplicates, init, add, sync, list_cmd # noqa: E402
42
+
43
+
44
+
if __name__ == "__main__":
45
+
app()
+125
src/thicket/cli/utils.py
+125
src/thicket/cli/utils.py
···
1
+
"""CLI utilities and helpers."""
2
+
3
+
from pathlib import Path
4
+
from typing import Optional
5
+
6
+
import typer
7
+
from rich.console import Console
8
+
from rich.progress import Progress, SpinnerColumn, TextColumn
9
+
from rich.table import Table
10
+
11
+
from ..models import ThicketConfig
12
+
13
+
console = Console()
14
+
15
+
16
+
def load_config(config_path: Optional[Path] = None) -> ThicketConfig:
17
+
"""Load thicket configuration from file or environment."""
18
+
if config_path and config_path.exists():
19
+
import yaml
20
+
21
+
with open(config_path) as f:
22
+
config_data = yaml.safe_load(f)
23
+
24
+
# Convert to ThicketConfig
25
+
return ThicketConfig(**config_data)
26
+
27
+
# Try to load from default locations or environment
28
+
try:
29
+
return ThicketConfig()
30
+
except Exception as e:
31
+
console.print(f"[red]Error loading configuration: {e}[/red]")
32
+
console.print("[yellow]Run 'thicket init' to create a new configuration.[/yellow]")
33
+
raise typer.Exit(1)
34
+
35
+
36
+
def save_config(config: ThicketConfig, config_path: Path) -> None:
37
+
"""Save thicket configuration to file."""
38
+
import yaml
39
+
40
+
config_data = config.model_dump(mode="json")
41
+
42
+
# Convert Path objects to strings for YAML serialization
43
+
config_data["git_store"] = str(config_data["git_store"])
44
+
config_data["cache_dir"] = str(config_data["cache_dir"])
45
+
46
+
with open(config_path, "w") as f:
47
+
yaml.dump(config_data, f, default_flow_style=False, sort_keys=False)
48
+
49
+
50
+
def create_progress() -> Progress:
51
+
"""Create a Rich progress display."""
52
+
return Progress(
53
+
SpinnerColumn(),
54
+
TextColumn("[progress.description]{task.description}"),
55
+
console=console,
56
+
transient=True,
57
+
)
58
+
59
+
60
+
def print_users_table(config: ThicketConfig) -> None:
61
+
"""Print a table of users and their feeds."""
62
+
table = Table(title="Users and Feeds")
63
+
table.add_column("Username", style="cyan", no_wrap=True)
64
+
table.add_column("Display Name", style="magenta")
65
+
table.add_column("Email", style="blue")
66
+
table.add_column("Homepage", style="green")
67
+
table.add_column("Feeds", style="yellow")
68
+
69
+
for user in config.users:
70
+
feeds_str = "\n".join(str(feed) for feed in user.feeds)
71
+
table.add_row(
72
+
user.username,
73
+
user.display_name or "",
74
+
user.email or "",
75
+
str(user.homepage) if user.homepage else "",
76
+
feeds_str,
77
+
)
78
+
79
+
console.print(table)
80
+
81
+
82
+
def print_feeds_table(config: ThicketConfig, username: Optional[str] = None) -> None:
83
+
"""Print a table of feeds, optionally filtered by username."""
84
+
table = Table(title=f"Feeds{f' for {username}' if username else ''}")
85
+
table.add_column("Username", style="cyan", no_wrap=True)
86
+
table.add_column("Feed URL", style="blue")
87
+
table.add_column("Status", style="green")
88
+
89
+
users = [config.find_user(username)] if username else config.users
90
+
users = [u for u in users if u is not None]
91
+
92
+
for user in users:
93
+
for feed in user.feeds:
94
+
table.add_row(
95
+
user.username,
96
+
str(feed),
97
+
"Active", # TODO: Add actual status checking
98
+
)
99
+
100
+
console.print(table)
101
+
102
+
103
+
def confirm_action(message: str, default: bool = False) -> bool:
104
+
"""Prompt for confirmation."""
105
+
return typer.confirm(message, default=default)
106
+
107
+
108
+
def print_success(message: str) -> None:
109
+
"""Print a success message."""
110
+
console.print(f"[green][/green] {message}")
111
+
112
+
113
+
def print_error(message: str) -> None:
114
+
"""Print an error message."""
115
+
console.print(f"[red][/red] {message}")
116
+
117
+
118
+
def print_warning(message: str) -> None:
119
+
"""Print a warning message."""
120
+
console.print(f"[yellow]�[/yellow] {message}")
121
+
122
+
123
+
def print_info(message: str) -> None:
124
+
"""Print an info message."""
125
+
console.print(f"[blue]9[/blue] {message}")
+6
src/thicket/core/__init__.py
+6
src/thicket/core/__init__.py
+262
src/thicket/core/feed_parser.py
+262
src/thicket/core/feed_parser.py
···
1
+
"""Feed parsing and normalization with auto-discovery."""
2
+
3
+
from datetime import datetime
4
+
from typing import Optional
5
+
from urllib.parse import urljoin, urlparse
6
+
7
+
import bleach
8
+
import feedparser
9
+
import httpx
10
+
from pydantic import HttpUrl, ValidationError
11
+
12
+
from ..models import AtomEntry, FeedMetadata
13
+
14
+
15
+
class FeedParser:
16
+
"""Parser for RSS/Atom feeds with normalization and auto-discovery."""
17
+
18
+
def __init__(self, user_agent: str = "thicket/0.1.0"):
19
+
"""Initialize the feed parser."""
20
+
self.user_agent = user_agent
21
+
self.allowed_tags = [
22
+
"a", "abbr", "acronym", "b", "blockquote", "br", "code", "em",
23
+
"i", "li", "ol", "p", "pre", "strong", "ul", "h1", "h2", "h3",
24
+
"h4", "h5", "h6", "img", "div", "span",
25
+
]
26
+
self.allowed_attributes = {
27
+
"a": ["href", "title"],
28
+
"abbr": ["title"],
29
+
"acronym": ["title"],
30
+
"img": ["src", "alt", "title", "width", "height"],
31
+
"blockquote": ["cite"],
32
+
}
33
+
34
+
async def fetch_feed(self, url: HttpUrl) -> str:
35
+
"""Fetch feed content from URL."""
36
+
async with httpx.AsyncClient() as client:
37
+
response = await client.get(
38
+
str(url),
39
+
headers={"User-Agent": self.user_agent},
40
+
timeout=30.0,
41
+
follow_redirects=True,
42
+
)
43
+
response.raise_for_status()
44
+
return response.text
45
+
46
+
def parse_feed(self, content: str, source_url: Optional[HttpUrl] = None) -> tuple[FeedMetadata, list[AtomEntry]]:
47
+
"""Parse feed content and return metadata and entries."""
48
+
parsed = feedparser.parse(content)
49
+
50
+
if parsed.bozo and parsed.bozo_exception:
51
+
# Try to continue with potentially malformed feed
52
+
pass
53
+
54
+
# Extract feed metadata
55
+
feed_meta = self._extract_feed_metadata(parsed.feed)
56
+
57
+
# Extract and normalize entries
58
+
entries = []
59
+
for entry in parsed.entries:
60
+
try:
61
+
atom_entry = self._normalize_entry(entry, source_url)
62
+
entries.append(atom_entry)
63
+
except Exception as e:
64
+
# Log error but continue processing other entries
65
+
print(f"Error processing entry {getattr(entry, 'id', 'unknown')}: {e}")
66
+
continue
67
+
68
+
return feed_meta, entries
69
+
70
+
def _extract_feed_metadata(self, feed: feedparser.FeedParserDict) -> FeedMetadata:
71
+
"""Extract metadata from feed for auto-discovery."""
72
+
# Parse author information
73
+
author_name = None
74
+
author_email = None
75
+
author_uri = None
76
+
77
+
if hasattr(feed, 'author_detail'):
78
+
author_name = feed.author_detail.get('name')
79
+
author_email = feed.author_detail.get('email')
80
+
author_uri = feed.author_detail.get('href')
81
+
elif hasattr(feed, 'author'):
82
+
author_name = feed.author
83
+
84
+
# Parse managing editor for RSS feeds
85
+
if not author_email and hasattr(feed, 'managingEditor'):
86
+
author_email = feed.managingEditor
87
+
88
+
# Parse feed link
89
+
feed_link = None
90
+
if hasattr(feed, 'link'):
91
+
try:
92
+
feed_link = HttpUrl(feed.link)
93
+
except ValidationError:
94
+
pass
95
+
96
+
# Parse image/icon/logo
97
+
logo = None
98
+
icon = None
99
+
image_url = None
100
+
101
+
if hasattr(feed, 'image'):
102
+
try:
103
+
image_url = HttpUrl(feed.image.get('href', feed.image.get('url', '')))
104
+
except (ValidationError, AttributeError):
105
+
pass
106
+
107
+
if hasattr(feed, 'icon'):
108
+
try:
109
+
icon = HttpUrl(feed.icon)
110
+
except ValidationError:
111
+
pass
112
+
113
+
if hasattr(feed, 'logo'):
114
+
try:
115
+
logo = HttpUrl(feed.logo)
116
+
except ValidationError:
117
+
pass
118
+
119
+
return FeedMetadata(
120
+
title=getattr(feed, 'title', None),
121
+
author_name=author_name,
122
+
author_email=author_email,
123
+
author_uri=HttpUrl(author_uri) if author_uri else None,
124
+
link=feed_link,
125
+
logo=logo,
126
+
icon=icon,
127
+
image_url=image_url,
128
+
description=getattr(feed, 'description', None),
129
+
)
130
+
131
+
def _normalize_entry(self, entry: feedparser.FeedParserDict, source_url: Optional[HttpUrl] = None) -> AtomEntry:
132
+
"""Normalize an entry to Atom format."""
133
+
# Parse timestamps
134
+
updated = self._parse_timestamp(entry.get('updated_parsed') or entry.get('published_parsed'))
135
+
published = self._parse_timestamp(entry.get('published_parsed'))
136
+
137
+
# Parse content
138
+
content = self._extract_content(entry)
139
+
content_type = self._extract_content_type(entry)
140
+
141
+
# Parse author
142
+
author = self._extract_author(entry)
143
+
144
+
# Parse categories/tags
145
+
categories = []
146
+
if hasattr(entry, 'tags'):
147
+
categories = [tag.get('term', '') for tag in entry.tags if tag.get('term')]
148
+
149
+
# Sanitize HTML content
150
+
if content:
151
+
content = self._sanitize_html(content)
152
+
153
+
summary = entry.get('summary', '')
154
+
if summary:
155
+
summary = self._sanitize_html(summary)
156
+
157
+
return AtomEntry(
158
+
id=entry.get('id', entry.get('link', '')),
159
+
title=entry.get('title', ''),
160
+
link=HttpUrl(entry.get('link', '')),
161
+
updated=updated,
162
+
published=published,
163
+
summary=summary or None,
164
+
content=content or None,
165
+
content_type=content_type,
166
+
author=author,
167
+
categories=categories,
168
+
rights=entry.get('rights', None),
169
+
source=str(source_url) if source_url else None,
170
+
)
171
+
172
+
def _parse_timestamp(self, time_struct) -> datetime:
173
+
"""Parse feedparser time struct to datetime."""
174
+
if time_struct:
175
+
return datetime(*time_struct[:6])
176
+
return datetime.now()
177
+
178
+
def _extract_content(self, entry: feedparser.FeedParserDict) -> Optional[str]:
179
+
"""Extract the best content from an entry."""
180
+
# Prefer content over summary
181
+
if hasattr(entry, 'content') and entry.content:
182
+
# Find the best content (prefer text/html, then text/plain)
183
+
for content_item in entry.content:
184
+
if content_item.get('type') in ['text/html', 'html']:
185
+
return content_item.get('value', '')
186
+
elif content_item.get('type') in ['text/plain', 'text']:
187
+
return content_item.get('value', '')
188
+
# Fallback to first content item
189
+
return entry.content[0].get('value', '')
190
+
191
+
# Fallback to summary
192
+
return entry.get('summary', '')
193
+
194
+
def _extract_content_type(self, entry: feedparser.FeedParserDict) -> str:
195
+
"""Extract content type from entry."""
196
+
if hasattr(entry, 'content') and entry.content:
197
+
content_type = entry.content[0].get('type', 'html')
198
+
# Normalize content type
199
+
if content_type in ['text/html', 'html']:
200
+
return 'html'
201
+
elif content_type in ['text/plain', 'text']:
202
+
return 'text'
203
+
elif content_type == 'xhtml':
204
+
return 'xhtml'
205
+
return 'html'
206
+
207
+
def _extract_author(self, entry: feedparser.FeedParserDict) -> Optional[dict]:
208
+
"""Extract author information from entry."""
209
+
author = {}
210
+
211
+
if hasattr(entry, 'author_detail'):
212
+
author.update({
213
+
'name': entry.author_detail.get('name'),
214
+
'email': entry.author_detail.get('email'),
215
+
'uri': entry.author_detail.get('href'),
216
+
})
217
+
elif hasattr(entry, 'author'):
218
+
author['name'] = entry.author
219
+
220
+
return author if author else None
221
+
222
+
def _sanitize_html(self, html: str) -> str:
223
+
"""Sanitize HTML content to prevent XSS."""
224
+
return bleach.clean(
225
+
html,
226
+
tags=self.allowed_tags,
227
+
attributes=self.allowed_attributes,
228
+
strip=True,
229
+
)
230
+
231
+
def sanitize_entry_id(self, entry_id: str) -> str:
232
+
"""Sanitize entry ID to be a safe filename."""
233
+
# Parse URL to get meaningful parts
234
+
parsed = urlparse(entry_id)
235
+
236
+
# Start with the path component
237
+
if parsed.path:
238
+
# Remove leading slash and replace problematic characters
239
+
safe_id = parsed.path.lstrip('/').replace('/', '_').replace('\\', '_')
240
+
else:
241
+
# Use the entire ID as fallback
242
+
safe_id = entry_id
243
+
244
+
# Replace problematic characters
245
+
safe_chars = []
246
+
for char in safe_id:
247
+
if char.isalnum() or char in '-_.':
248
+
safe_chars.append(char)
249
+
else:
250
+
safe_chars.append('_')
251
+
252
+
safe_id = ''.join(safe_chars)
253
+
254
+
# Ensure it's not too long (max 200 chars)
255
+
if len(safe_id) > 200:
256
+
safe_id = safe_id[:200]
257
+
258
+
# Ensure it's not empty
259
+
if not safe_id:
260
+
safe_id = "entry"
261
+
262
+
return safe_id
+321
src/thicket/core/git_store.py
+321
src/thicket/core/git_store.py
···
1
+
"""Git repository operations for thicket."""
2
+
3
+
import json
4
+
from datetime import datetime
5
+
from pathlib import Path
6
+
from typing import Optional
7
+
8
+
import git
9
+
from git import Repo
10
+
11
+
from ..models import AtomEntry, DuplicateMap, GitStoreIndex, UserMetadata
12
+
13
+
14
+
class GitStore:
15
+
"""Manages the Git repository for storing feed entries."""
16
+
17
+
def __init__(self, repo_path: Path):
18
+
"""Initialize the Git store."""
19
+
self.repo_path = repo_path
20
+
self.repo: Optional[Repo] = None
21
+
self._ensure_repo()
22
+
23
+
def _ensure_repo(self) -> None:
24
+
"""Ensure the Git repository exists and is initialized."""
25
+
if not self.repo_path.exists():
26
+
self.repo_path.mkdir(parents=True, exist_ok=True)
27
+
28
+
try:
29
+
self.repo = Repo(self.repo_path)
30
+
except git.InvalidGitRepositoryError:
31
+
# Initialize new repository
32
+
self.repo = Repo.init(self.repo_path)
33
+
self._create_initial_structure()
34
+
35
+
def _create_initial_structure(self) -> None:
36
+
"""Create initial Git store structure."""
37
+
# Create index.json
38
+
index = GitStoreIndex(
39
+
created=datetime.now(),
40
+
last_updated=datetime.now(),
41
+
)
42
+
self._save_index(index)
43
+
44
+
# Create duplicates.json
45
+
duplicates = DuplicateMap()
46
+
self._save_duplicates(duplicates)
47
+
48
+
# Create initial commit
49
+
self.repo.index.add(["index.json", "duplicates.json"])
50
+
self.repo.index.commit("Initial thicket repository structure")
51
+
52
+
def _save_index(self, index: GitStoreIndex) -> None:
53
+
"""Save the index to index.json."""
54
+
index_path = self.repo_path / "index.json"
55
+
with open(index_path, "w") as f:
56
+
json.dump(index.model_dump(mode="json"), f, indent=2, default=str)
57
+
58
+
def _load_index(self) -> GitStoreIndex:
59
+
"""Load the index from index.json."""
60
+
index_path = self.repo_path / "index.json"
61
+
if not index_path.exists():
62
+
return GitStoreIndex(
63
+
created=datetime.now(),
64
+
last_updated=datetime.now(),
65
+
)
66
+
67
+
with open(index_path) as f:
68
+
data = json.load(f)
69
+
70
+
return GitStoreIndex(**data)
71
+
72
+
def _save_duplicates(self, duplicates: DuplicateMap) -> None:
73
+
"""Save duplicates map to duplicates.json."""
74
+
duplicates_path = self.repo_path / "duplicates.json"
75
+
with open(duplicates_path, "w") as f:
76
+
json.dump(duplicates.model_dump(), f, indent=2)
77
+
78
+
def _load_duplicates(self) -> DuplicateMap:
79
+
"""Load duplicates map from duplicates.json."""
80
+
duplicates_path = self.repo_path / "duplicates.json"
81
+
if not duplicates_path.exists():
82
+
return DuplicateMap()
83
+
84
+
with open(duplicates_path) as f:
85
+
data = json.load(f)
86
+
87
+
return DuplicateMap(**data)
88
+
89
+
def add_user(self, username: str, display_name: Optional[str] = None,
90
+
email: Optional[str] = None, homepage: Optional[str] = None,
91
+
icon: Optional[str] = None, feeds: Optional[list[str]] = None) -> UserMetadata:
92
+
"""Add a new user to the Git store."""
93
+
index = self._load_index()
94
+
95
+
# Create user directory
96
+
user_dir = self.repo_path / username
97
+
user_dir.mkdir(exist_ok=True)
98
+
99
+
# Create user metadata
100
+
user_metadata = UserMetadata(
101
+
username=username,
102
+
display_name=display_name,
103
+
email=email,
104
+
homepage=homepage,
105
+
icon=icon,
106
+
feeds=feeds or [],
107
+
directory=username,
108
+
created=datetime.now(),
109
+
last_updated=datetime.now(),
110
+
)
111
+
112
+
# Save user metadata
113
+
metadata_path = user_dir / "metadata.json"
114
+
with open(metadata_path, "w") as f:
115
+
json.dump(user_metadata.model_dump(mode="json"), f, indent=2, default=str)
116
+
117
+
# Update index
118
+
index.add_user(user_metadata)
119
+
self._save_index(index)
120
+
121
+
return user_metadata
122
+
123
+
def get_user(self, username: str) -> Optional[UserMetadata]:
124
+
"""Get user metadata by username."""
125
+
index = self._load_index()
126
+
return index.get_user(username)
127
+
128
+
def update_user(self, username: str, **kwargs) -> bool:
129
+
"""Update user metadata."""
130
+
index = self._load_index()
131
+
user = index.get_user(username)
132
+
133
+
if not user:
134
+
return False
135
+
136
+
# Update user metadata
137
+
for key, value in kwargs.items():
138
+
if hasattr(user, key) and value is not None:
139
+
setattr(user, key, value)
140
+
141
+
user.update_timestamp()
142
+
143
+
# Save user metadata
144
+
user_dir = self.repo_path / user.directory
145
+
metadata_path = user_dir / "metadata.json"
146
+
with open(metadata_path, "w") as f:
147
+
json.dump(user.model_dump(mode="json"), f, indent=2, default=str)
148
+
149
+
# Update index
150
+
index.add_user(user)
151
+
self._save_index(index)
152
+
153
+
return True
154
+
155
+
def store_entry(self, username: str, entry: AtomEntry) -> bool:
156
+
"""Store an entry in the user's directory."""
157
+
user = self.get_user(username)
158
+
if not user:
159
+
return False
160
+
161
+
# Sanitize entry ID for filename
162
+
from .feed_parser import FeedParser
163
+
parser = FeedParser()
164
+
safe_id = parser.sanitize_entry_id(entry.id)
165
+
166
+
# Create entry file
167
+
user_dir = self.repo_path / user.directory
168
+
entry_path = user_dir / f"{safe_id}.json"
169
+
170
+
# Check if entry already exists
171
+
entry_exists = entry_path.exists()
172
+
173
+
# Save entry
174
+
with open(entry_path, "w") as f:
175
+
json.dump(entry.model_dump(mode="json"), f, indent=2, default=str)
176
+
177
+
# Update user metadata if new entry
178
+
if not entry_exists:
179
+
user.increment_entry_count()
180
+
self.update_user(username, entry_count=user.entry_count)
181
+
182
+
return True
183
+
184
+
def get_entry(self, username: str, entry_id: str) -> Optional[AtomEntry]:
185
+
"""Get an entry by username and entry ID."""
186
+
user = self.get_user(username)
187
+
if not user:
188
+
return None
189
+
190
+
# Sanitize entry ID
191
+
from .feed_parser import FeedParser
192
+
parser = FeedParser()
193
+
safe_id = parser.sanitize_entry_id(entry_id)
194
+
195
+
entry_path = self.repo_path / user.directory / f"{safe_id}.json"
196
+
if not entry_path.exists():
197
+
return None
198
+
199
+
with open(entry_path) as f:
200
+
data = json.load(f)
201
+
202
+
return AtomEntry(**data)
203
+
204
+
def list_entries(self, username: str, limit: Optional[int] = None) -> list[AtomEntry]:
205
+
"""List entries for a user."""
206
+
user = self.get_user(username)
207
+
if not user:
208
+
return []
209
+
210
+
user_dir = self.repo_path / user.directory
211
+
if not user_dir.exists():
212
+
return []
213
+
214
+
entries = []
215
+
entry_files = sorted(user_dir.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True)
216
+
217
+
# Filter out metadata.json
218
+
entry_files = [f for f in entry_files if f.name != "metadata.json"]
219
+
220
+
if limit:
221
+
entry_files = entry_files[:limit]
222
+
223
+
for entry_file in entry_files:
224
+
try:
225
+
with open(entry_file) as f:
226
+
data = json.load(f)
227
+
entries.append(AtomEntry(**data))
228
+
except Exception:
229
+
# Skip invalid entries
230
+
continue
231
+
232
+
return entries
233
+
234
+
def get_duplicates(self) -> DuplicateMap:
235
+
"""Get the duplicates map."""
236
+
return self._load_duplicates()
237
+
238
+
def add_duplicate(self, duplicate_id: str, canonical_id: str) -> None:
239
+
"""Add a duplicate mapping."""
240
+
duplicates = self._load_duplicates()
241
+
duplicates.add_duplicate(duplicate_id, canonical_id)
242
+
self._save_duplicates(duplicates)
243
+
244
+
def remove_duplicate(self, duplicate_id: str) -> bool:
245
+
"""Remove a duplicate mapping."""
246
+
duplicates = self._load_duplicates()
247
+
result = duplicates.remove_duplicate(duplicate_id)
248
+
self._save_duplicates(duplicates)
249
+
return result
250
+
251
+
def commit_changes(self, message: str) -> None:
252
+
"""Commit all changes to the Git repository."""
253
+
if not self.repo:
254
+
return
255
+
256
+
# Add all changes
257
+
self.repo.git.add(A=True)
258
+
259
+
# Check if there are changes to commit
260
+
if self.repo.index.diff("HEAD"):
261
+
self.repo.index.commit(message)
262
+
263
+
def get_stats(self) -> dict:
264
+
"""Get statistics about the Git store."""
265
+
index = self._load_index()
266
+
duplicates = self._load_duplicates()
267
+
268
+
return {
269
+
"total_users": len(index.users),
270
+
"total_entries": index.total_entries,
271
+
"total_duplicates": len(duplicates.duplicates),
272
+
"last_updated": index.last_updated,
273
+
"repository_size": sum(f.stat().st_size for f in self.repo_path.rglob("*") if f.is_file()),
274
+
}
275
+
276
+
def search_entries(self, query: str, username: Optional[str] = None,
277
+
limit: Optional[int] = None) -> list[tuple[str, AtomEntry]]:
278
+
"""Search entries by content."""
279
+
results = []
280
+
281
+
# Get users to search
282
+
index = self._load_index()
283
+
users = [index.get_user(username)] if username else list(index.users.values())
284
+
users = [u for u in users if u is not None]
285
+
286
+
for user in users:
287
+
user_dir = self.repo_path / user.directory
288
+
if not user_dir.exists():
289
+
continue
290
+
291
+
entry_files = user_dir.glob("*.json")
292
+
entry_files = [f for f in entry_files if f.name != "metadata.json"]
293
+
294
+
for entry_file in entry_files:
295
+
try:
296
+
with open(entry_file) as f:
297
+
data = json.load(f)
298
+
299
+
entry = AtomEntry(**data)
300
+
301
+
# Simple text search in title, summary, and content
302
+
searchable_text = " ".join(filter(None, [
303
+
entry.title,
304
+
entry.summary or "",
305
+
entry.content or "",
306
+
])).lower()
307
+
308
+
if query.lower() in searchable_text:
309
+
results.append((user.username, entry))
310
+
311
+
if limit and len(results) >= limit:
312
+
return results
313
+
314
+
except Exception:
315
+
# Skip invalid entries
316
+
continue
317
+
318
+
# Sort by updated time (newest first)
319
+
results.sort(key=lambda x: x[1].updated, reverse=True)
320
+
321
+
return results[:limit] if limit else results
+15
src/thicket/models/__init__.py
+15
src/thicket/models/__init__.py
···
1
+
"""Data models for thicket."""
2
+
3
+
from .config import ThicketConfig, UserConfig
4
+
from .feed import AtomEntry, DuplicateMap, FeedMetadata
5
+
from .user import GitStoreIndex, UserMetadata
6
+
7
+
__all__ = [
8
+
"ThicketConfig",
9
+
"UserConfig",
10
+
"AtomEntry",
11
+
"DuplicateMap",
12
+
"FeedMetadata",
13
+
"GitStoreIndex",
14
+
"UserMetadata",
15
+
]
+71
src/thicket/models/config.py
+71
src/thicket/models/config.py
···
1
+
"""Configuration models for thicket."""
2
+
3
+
from pathlib import Path
4
+
from typing import Optional
5
+
6
+
from pydantic import BaseModel, EmailStr, HttpUrl
7
+
from pydantic_settings import BaseSettings, SettingsConfigDict
8
+
9
+
10
+
class UserConfig(BaseModel):
11
+
"""Configuration for a single user and their feeds."""
12
+
13
+
username: str
14
+
feeds: list[HttpUrl]
15
+
email: Optional[EmailStr] = None
16
+
homepage: Optional[HttpUrl] = None
17
+
icon: Optional[HttpUrl] = None
18
+
display_name: Optional[str] = None
19
+
20
+
21
+
class ThicketConfig(BaseSettings):
22
+
"""Main configuration for thicket."""
23
+
24
+
model_config = SettingsConfigDict(
25
+
env_prefix="THICKET_",
26
+
env_file=".env",
27
+
yaml_file="thicket.yaml",
28
+
case_sensitive=False,
29
+
)
30
+
31
+
git_store: Path
32
+
cache_dir: Path
33
+
users: list[UserConfig] = []
34
+
35
+
def find_user(self, username: str) -> Optional[UserConfig]:
36
+
"""Find a user by username."""
37
+
for user in self.users:
38
+
if user.username == username:
39
+
return user
40
+
return None
41
+
42
+
def add_user(self, user: UserConfig) -> None:
43
+
"""Add a new user or update existing user."""
44
+
existing = self.find_user(user.username)
45
+
if existing:
46
+
# Update existing user
47
+
existing.feeds = list(set(existing.feeds + user.feeds))
48
+
existing.email = user.email or existing.email
49
+
existing.homepage = user.homepage or existing.homepage
50
+
existing.icon = user.icon or existing.icon
51
+
existing.display_name = user.display_name or existing.display_name
52
+
else:
53
+
# Add new user
54
+
self.users.append(user)
55
+
56
+
def remove_user(self, username: str) -> bool:
57
+
"""Remove a user by username. Returns True if user was found and removed."""
58
+
for i, user in enumerate(self.users):
59
+
if user.username == username:
60
+
del self.users[i]
61
+
return True
62
+
return False
63
+
64
+
def add_feed_to_user(self, username: str, feed_url: HttpUrl) -> bool:
65
+
"""Add a feed to an existing user. Returns True if user was found."""
66
+
user = self.find_user(username)
67
+
if user:
68
+
if feed_url not in user.feeds:
69
+
user.feeds.append(feed_url)
70
+
return True
71
+
return False
+86
src/thicket/models/feed.py
+86
src/thicket/models/feed.py
···
1
+
"""Feed and entry models for thicket."""
2
+
3
+
from datetime import datetime
4
+
from typing import Optional
5
+
6
+
from pydantic import BaseModel, ConfigDict, EmailStr, HttpUrl
7
+
8
+
9
+
class AtomEntry(BaseModel):
10
+
"""Represents an Atom feed entry stored in the Git repository."""
11
+
12
+
model_config = ConfigDict(
13
+
json_encoders={datetime: lambda v: v.isoformat()},
14
+
str_strip_whitespace=True,
15
+
)
16
+
17
+
id: str # Original Atom ID
18
+
title: str
19
+
link: HttpUrl
20
+
updated: datetime
21
+
published: Optional[datetime] = None
22
+
summary: Optional[str] = None
23
+
content: Optional[str] = None # Full body content from Atom entry
24
+
content_type: Optional[str] = "html" # text, html, xhtml
25
+
author: Optional[dict] = None
26
+
categories: list[str] = []
27
+
rights: Optional[str] = None # Copyright info
28
+
source: Optional[str] = None # Source feed URL
29
+
30
+
31
+
class FeedMetadata(BaseModel):
32
+
"""Metadata extracted from a feed for auto-discovery."""
33
+
34
+
title: Optional[str] = None
35
+
author_name: Optional[str] = None
36
+
author_email: Optional[EmailStr] = None
37
+
author_uri: Optional[HttpUrl] = None
38
+
link: Optional[HttpUrl] = None
39
+
logo: Optional[HttpUrl] = None
40
+
icon: Optional[HttpUrl] = None
41
+
image_url: Optional[HttpUrl] = None
42
+
description: Optional[str] = None
43
+
44
+
def to_user_config(self, username: str, feed_url: HttpUrl) -> "UserConfig":
45
+
"""Convert discovered metadata to UserConfig with fallbacks."""
46
+
from .config import UserConfig
47
+
48
+
return UserConfig(
49
+
username=username,
50
+
feeds=[feed_url],
51
+
display_name=self.author_name or self.title,
52
+
email=self.author_email,
53
+
homepage=self.author_uri or self.link,
54
+
icon=self.logo or self.icon or self.image_url,
55
+
)
56
+
57
+
58
+
class DuplicateMap(BaseModel):
59
+
"""Maps duplicate entry IDs to canonical entry IDs."""
60
+
61
+
duplicates: dict[str, str] = {} # duplicate_id -> canonical_id
62
+
comment: str = "Entry IDs that map to the same canonical content"
63
+
64
+
def add_duplicate(self, duplicate_id: str, canonical_id: str) -> None:
65
+
"""Add a duplicate mapping."""
66
+
self.duplicates[duplicate_id] = canonical_id
67
+
68
+
def remove_duplicate(self, duplicate_id: str) -> bool:
69
+
"""Remove a duplicate mapping. Returns True if existed."""
70
+
return self.duplicates.pop(duplicate_id, None) is not None
71
+
72
+
def get_canonical(self, entry_id: str) -> str:
73
+
"""Get canonical ID for an entry (returns original if not duplicate)."""
74
+
return self.duplicates.get(entry_id, entry_id)
75
+
76
+
def is_duplicate(self, entry_id: str) -> bool:
77
+
"""Check if entry ID is marked as duplicate."""
78
+
return entry_id in self.duplicates
79
+
80
+
def get_duplicates_for_canonical(self, canonical_id: str) -> list[str]:
81
+
"""Get all duplicate IDs that map to a canonical ID."""
82
+
return [
83
+
duplicate_id
84
+
for duplicate_id, canonical in self.duplicates.items()
85
+
if canonical == canonical_id
86
+
]
+79
src/thicket/models/user.py
+79
src/thicket/models/user.py
···
1
+
"""User metadata models for thicket."""
2
+
3
+
from datetime import datetime
4
+
from typing import Optional
5
+
6
+
from pydantic import BaseModel, ConfigDict
7
+
8
+
9
+
class UserMetadata(BaseModel):
10
+
"""Metadata about a user stored in the Git repository."""
11
+
12
+
model_config = ConfigDict(
13
+
json_encoders={datetime: lambda v: v.isoformat()},
14
+
str_strip_whitespace=True,
15
+
)
16
+
17
+
username: str
18
+
display_name: Optional[str] = None
19
+
email: Optional[str] = None
20
+
homepage: Optional[str] = None
21
+
icon: Optional[str] = None
22
+
feeds: list[str] = []
23
+
directory: str # Directory name in Git store
24
+
created: datetime
25
+
last_updated: datetime
26
+
entry_count: int = 0
27
+
28
+
def update_timestamp(self) -> None:
29
+
"""Update the last_updated timestamp to now."""
30
+
self.last_updated = datetime.now()
31
+
32
+
def increment_entry_count(self, count: int = 1) -> None:
33
+
"""Increment the entry count by the given amount."""
34
+
self.entry_count += count
35
+
self.update_timestamp()
36
+
37
+
38
+
class GitStoreIndex(BaseModel):
39
+
"""Index of all users and their directories in the Git store."""
40
+
41
+
model_config = ConfigDict(
42
+
json_encoders={datetime: lambda v: v.isoformat()}
43
+
)
44
+
45
+
users: dict[str, UserMetadata] = {} # username -> UserMetadata
46
+
created: datetime
47
+
last_updated: datetime
48
+
total_entries: int = 0
49
+
50
+
def add_user(self, user_metadata: UserMetadata) -> None:
51
+
"""Add or update a user in the index."""
52
+
self.users[user_metadata.username] = user_metadata
53
+
self.last_updated = datetime.now()
54
+
55
+
def remove_user(self, username: str) -> bool:
56
+
"""Remove a user from the index. Returns True if user existed."""
57
+
if username in self.users:
58
+
del self.users[username]
59
+
self.last_updated = datetime.now()
60
+
return True
61
+
return False
62
+
63
+
def get_user(self, username: str) -> Optional[UserMetadata]:
64
+
"""Get user metadata by username."""
65
+
return self.users.get(username)
66
+
67
+
def update_entry_count(self, username: str, count: int) -> None:
68
+
"""Update entry count for a user and total."""
69
+
user = self.get_user(username)
70
+
if user:
71
+
old_count = user.entry_count
72
+
user.increment_entry_count(count)
73
+
self.total_entries += count
74
+
self.last_updated = datetime.now()
75
+
76
+
def recalculate_totals(self) -> None:
77
+
"""Recalculate total entries from all users."""
78
+
self.total_entries = sum(user.entry_count for user in self.users.values())
79
+
self.last_updated = datetime.now()
+4
src/thicket/utils/__init__.py
+4
src/thicket/utils/__init__.py
tests/__init__.py
tests/__init__.py
This is a binary file and will not be displayed.
+84
tests/conftest.py
+84
tests/conftest.py
···
1
+
"""Test configuration and fixtures for thicket."""
2
+
3
+
import tempfile
4
+
from pathlib import Path
5
+
6
+
import pytest
7
+
8
+
from thicket.models import ThicketConfig, UserConfig
9
+
10
+
11
+
@pytest.fixture
12
+
def temp_dir():
13
+
"""Create a temporary directory for tests."""
14
+
with tempfile.TemporaryDirectory() as tmp_dir:
15
+
yield Path(tmp_dir)
16
+
17
+
18
+
@pytest.fixture
19
+
def sample_config(temp_dir):
20
+
"""Create a sample configuration for testing."""
21
+
git_store = temp_dir / "git_store"
22
+
cache_dir = temp_dir / "cache"
23
+
24
+
return ThicketConfig(
25
+
git_store=git_store,
26
+
cache_dir=cache_dir,
27
+
users=[
28
+
UserConfig(
29
+
username="testuser",
30
+
feeds=["https://example.com/feed.xml"],
31
+
email="test@example.com",
32
+
display_name="Test User",
33
+
)
34
+
],
35
+
)
36
+
37
+
38
+
@pytest.fixture
39
+
def sample_atom_feed():
40
+
"""Sample Atom feed XML for testing."""
41
+
return """<?xml version="1.0" encoding="utf-8"?>
42
+
<feed xmlns="http://www.w3.org/2005/Atom">
43
+
<title>Test Feed</title>
44
+
<link href="https://example.com/"/>
45
+
<updated>2025-01-01T00:00:00Z</updated>
46
+
<author>
47
+
<name>Test Author</name>
48
+
<email>author@example.com</email>
49
+
</author>
50
+
<id>https://example.com/</id>
51
+
52
+
<entry>
53
+
<title>Test Entry</title>
54
+
<link href="https://example.com/entry/1"/>
55
+
<id>https://example.com/entry/1</id>
56
+
<updated>2025-01-01T00:00:00Z</updated>
57
+
<summary>This is a test entry.</summary>
58
+
<content type="html">
59
+
<![CDATA[<p>This is the content of the test entry.</p>]]>
60
+
</content>
61
+
</entry>
62
+
</feed>"""
63
+
64
+
65
+
@pytest.fixture
66
+
def sample_rss_feed():
67
+
"""Sample RSS feed XML for testing."""
68
+
return """<?xml version="1.0" encoding="UTF-8"?>
69
+
<rss version="2.0">
70
+
<channel>
71
+
<title>Test RSS Feed</title>
72
+
<link>https://example.com/</link>
73
+
<description>Test RSS feed for testing</description>
74
+
<managingEditor>editor@example.com</managingEditor>
75
+
76
+
<item>
77
+
<title>Test RSS Entry</title>
78
+
<link>https://example.com/rss/entry/1</link>
79
+
<description>This is a test RSS entry.</description>
80
+
<pubDate>Mon, 01 Jan 2025 00:00:00 GMT</pubDate>
81
+
<guid>https://example.com/rss/entry/1</guid>
82
+
</item>
83
+
</channel>
84
+
</rss>"""
+132
tests/test_feed_parser.py
+132
tests/test_feed_parser.py
···
1
+
"""Tests for feed parser functionality."""
2
+
3
+
import pytest
4
+
from pydantic import HttpUrl
5
+
6
+
from thicket.core.feed_parser import FeedParser
7
+
from thicket.models import AtomEntry, FeedMetadata
8
+
9
+
10
+
class TestFeedParser:
11
+
"""Test the FeedParser class."""
12
+
13
+
def test_init(self):
14
+
"""Test parser initialization."""
15
+
parser = FeedParser()
16
+
assert parser.user_agent == "thicket/0.1.0"
17
+
assert "a" in parser.allowed_tags
18
+
assert "href" in parser.allowed_attributes["a"]
19
+
20
+
def test_parse_atom_feed(self, sample_atom_feed):
21
+
"""Test parsing an Atom feed."""
22
+
parser = FeedParser()
23
+
metadata, entries = parser.parse_feed(sample_atom_feed)
24
+
25
+
# Check metadata
26
+
assert isinstance(metadata, FeedMetadata)
27
+
assert metadata.title == "Test Feed"
28
+
assert metadata.author_name == "Test Author"
29
+
assert metadata.author_email == "author@example.com"
30
+
assert metadata.link == HttpUrl("https://example.com/")
31
+
32
+
# Check entries
33
+
assert len(entries) == 1
34
+
entry = entries[0]
35
+
assert isinstance(entry, AtomEntry)
36
+
assert entry.title == "Test Entry"
37
+
assert entry.id == "https://example.com/entry/1"
38
+
assert entry.link == HttpUrl("https://example.com/entry/1")
39
+
assert entry.summary == "This is a test entry."
40
+
assert "<p>This is the content of the test entry.</p>" in entry.content
41
+
42
+
def test_parse_rss_feed(self, sample_rss_feed):
43
+
"""Test parsing an RSS feed."""
44
+
parser = FeedParser()
45
+
metadata, entries = parser.parse_feed(sample_rss_feed)
46
+
47
+
# Check metadata
48
+
assert isinstance(metadata, FeedMetadata)
49
+
assert metadata.title == "Test RSS Feed"
50
+
assert metadata.link == HttpUrl("https://example.com/")
51
+
assert metadata.author_email == "editor@example.com"
52
+
53
+
# Check entries
54
+
assert len(entries) == 1
55
+
entry = entries[0]
56
+
assert isinstance(entry, AtomEntry)
57
+
assert entry.title == "Test RSS Entry"
58
+
assert entry.id == "https://example.com/rss/entry/1"
59
+
assert entry.summary == "This is a test RSS entry."
60
+
61
+
def test_sanitize_entry_id(self):
62
+
"""Test entry ID sanitization."""
63
+
parser = FeedParser()
64
+
65
+
# Test URL ID
66
+
url_id = "https://example.com/posts/2025/01/test-post"
67
+
sanitized = parser.sanitize_entry_id(url_id)
68
+
assert sanitized == "posts_2025_01_test-post"
69
+
70
+
# Test problematic characters
71
+
bad_id = "test/with\\bad:chars|and<more>"
72
+
sanitized = parser.sanitize_entry_id(bad_id)
73
+
assert sanitized == "test_with_bad_chars_and_more_"
74
+
75
+
# Test empty ID
76
+
empty_id = ""
77
+
sanitized = parser.sanitize_entry_id(empty_id)
78
+
assert sanitized == "entry"
79
+
80
+
# Test very long ID
81
+
long_id = "a" * 300
82
+
sanitized = parser.sanitize_entry_id(long_id)
83
+
assert len(sanitized) == 200
84
+
85
+
def test_sanitize_html(self):
86
+
"""Test HTML sanitization."""
87
+
parser = FeedParser()
88
+
89
+
# Test allowed tags
90
+
safe_html = "<p>This is <strong>safe</strong> HTML</p>"
91
+
sanitized = parser._sanitize_html(safe_html)
92
+
assert sanitized == safe_html
93
+
94
+
# Test dangerous tags
95
+
dangerous_html = "<script>alert('xss')</script><p>Safe content</p>"
96
+
sanitized = parser._sanitize_html(dangerous_html)
97
+
assert "<script>" not in sanitized
98
+
assert "<p>Safe content</p>" in sanitized
99
+
100
+
# Test attributes
101
+
html_with_attrs = '<a href="https://example.com" onclick="alert()">Link</a>'
102
+
sanitized = parser._sanitize_html(html_with_attrs)
103
+
assert 'href="https://example.com"' in sanitized
104
+
assert 'onclick' not in sanitized
105
+
106
+
def test_extract_feed_metadata(self):
107
+
"""Test feed metadata extraction."""
108
+
parser = FeedParser()
109
+
110
+
# Test with feedparser parsed data
111
+
import feedparser
112
+
parsed = feedparser.parse("""<?xml version="1.0" encoding="utf-8"?>
113
+
<feed xmlns="http://www.w3.org/2005/Atom">
114
+
<title>Test Feed</title>
115
+
<link href="https://example.com/"/>
116
+
<author>
117
+
<name>Test Author</name>
118
+
<email>author@example.com</email>
119
+
<uri>https://example.com/about</uri>
120
+
</author>
121
+
<logo>https://example.com/logo.png</logo>
122
+
<icon>https://example.com/icon.png</icon>
123
+
</feed>""")
124
+
125
+
metadata = parser._extract_feed_metadata(parsed.feed)
126
+
assert metadata.title == "Test Feed"
127
+
assert metadata.author_name == "Test Author"
128
+
assert metadata.author_email == "author@example.com"
129
+
assert metadata.author_uri == HttpUrl("https://example.com/about")
130
+
assert metadata.link == HttpUrl("https://example.com/")
131
+
assert metadata.logo == HttpUrl("https://example.com/logo.png")
132
+
assert metadata.icon == HttpUrl("https://example.com/icon.png")
+277
tests/test_git_store.py
+277
tests/test_git_store.py
···
1
+
"""Tests for Git store functionality."""
2
+
3
+
import json
4
+
from datetime import datetime
5
+
6
+
import pytest
7
+
from pydantic import HttpUrl
8
+
9
+
from thicket.core.git_store import GitStore
10
+
from thicket.models import AtomEntry, DuplicateMap, UserMetadata
11
+
12
+
13
+
class TestGitStore:
14
+
"""Test the GitStore class."""
15
+
16
+
def test_init_new_repo(self, temp_dir):
17
+
"""Test initializing a new Git repository."""
18
+
repo_path = temp_dir / "test_repo"
19
+
store = GitStore(repo_path)
20
+
21
+
assert store.repo_path == repo_path
22
+
assert store.repo is not None
23
+
assert repo_path.exists()
24
+
assert (repo_path / ".git").exists()
25
+
assert (repo_path / "index.json").exists()
26
+
assert (repo_path / "duplicates.json").exists()
27
+
28
+
def test_init_existing_repo(self, temp_dir):
29
+
"""Test initializing with existing repository."""
30
+
repo_path = temp_dir / "test_repo"
31
+
32
+
# Create first store
33
+
store1 = GitStore(repo_path)
34
+
store1.add_user("testuser", display_name="Test User")
35
+
36
+
# Create second store pointing to same repo
37
+
store2 = GitStore(repo_path)
38
+
user = store2.get_user("testuser")
39
+
40
+
assert user is not None
41
+
assert user.username == "testuser"
42
+
assert user.display_name == "Test User"
43
+
44
+
def test_add_user(self, temp_dir):
45
+
"""Test adding a user to the Git store."""
46
+
store = GitStore(temp_dir / "test_repo")
47
+
48
+
user = store.add_user(
49
+
username="testuser",
50
+
display_name="Test User",
51
+
email="test@example.com",
52
+
homepage="https://example.com",
53
+
icon="https://example.com/icon.png",
54
+
feeds=["https://example.com/feed.xml"],
55
+
)
56
+
57
+
assert isinstance(user, UserMetadata)
58
+
assert user.username == "testuser"
59
+
assert user.display_name == "Test User"
60
+
assert user.email == "test@example.com"
61
+
assert user.homepage == "https://example.com"
62
+
assert user.icon == "https://example.com/icon.png"
63
+
assert user.feeds == ["https://example.com/feed.xml"]
64
+
assert user.directory == "testuser"
65
+
66
+
# Check that user directory was created
67
+
user_dir = store.repo_path / "testuser"
68
+
assert user_dir.exists()
69
+
assert (user_dir / "metadata.json").exists()
70
+
71
+
# Check metadata file content
72
+
with open(user_dir / "metadata.json") as f:
73
+
metadata = json.load(f)
74
+
assert metadata["username"] == "testuser"
75
+
assert metadata["display_name"] == "Test User"
76
+
77
+
def test_get_user(self, temp_dir):
78
+
"""Test getting user metadata."""
79
+
store = GitStore(temp_dir / "test_repo")
80
+
81
+
# Add user
82
+
store.add_user("testuser", display_name="Test User")
83
+
84
+
# Get user
85
+
user = store.get_user("testuser")
86
+
assert user is not None
87
+
assert user.username == "testuser"
88
+
assert user.display_name == "Test User"
89
+
90
+
# Try to get non-existent user
91
+
non_user = store.get_user("nonexistent")
92
+
assert non_user is None
93
+
94
+
def test_store_entry(self, temp_dir):
95
+
"""Test storing an entry."""
96
+
store = GitStore(temp_dir / "test_repo")
97
+
98
+
# Add user first
99
+
store.add_user("testuser")
100
+
101
+
# Create test entry
102
+
entry = AtomEntry(
103
+
id="https://example.com/entry/1",
104
+
title="Test Entry",
105
+
link=HttpUrl("https://example.com/entry/1"),
106
+
updated=datetime.now(),
107
+
summary="Test entry summary",
108
+
content="<p>Test content</p>",
109
+
)
110
+
111
+
# Store entry
112
+
result = store.store_entry("testuser", entry)
113
+
assert result is True
114
+
115
+
# Check that entry file was created
116
+
user_dir = store.repo_path / "testuser"
117
+
entry_files = list(user_dir.glob("*.json"))
118
+
entry_files = [f for f in entry_files if f.name != "metadata.json"]
119
+
assert len(entry_files) == 1
120
+
121
+
# Check entry content
122
+
with open(entry_files[0]) as f:
123
+
stored_entry = json.load(f)
124
+
assert stored_entry["title"] == "Test Entry"
125
+
assert stored_entry["id"] == "https://example.com/entry/1"
126
+
127
+
def test_get_entry(self, temp_dir):
128
+
"""Test retrieving an entry."""
129
+
store = GitStore(temp_dir / "test_repo")
130
+
131
+
# Add user and entry
132
+
store.add_user("testuser")
133
+
entry = AtomEntry(
134
+
id="https://example.com/entry/1",
135
+
title="Test Entry",
136
+
link=HttpUrl("https://example.com/entry/1"),
137
+
updated=datetime.now(),
138
+
)
139
+
store.store_entry("testuser", entry)
140
+
141
+
# Get entry
142
+
retrieved = store.get_entry("testuser", "https://example.com/entry/1")
143
+
assert retrieved is not None
144
+
assert retrieved.title == "Test Entry"
145
+
assert retrieved.id == "https://example.com/entry/1"
146
+
147
+
# Try to get non-existent entry
148
+
non_entry = store.get_entry("testuser", "https://example.com/nonexistent")
149
+
assert non_entry is None
150
+
151
+
def test_list_entries(self, temp_dir):
152
+
"""Test listing entries for a user."""
153
+
store = GitStore(temp_dir / "test_repo")
154
+
155
+
# Add user
156
+
store.add_user("testuser")
157
+
158
+
# Add multiple entries
159
+
for i in range(3):
160
+
entry = AtomEntry(
161
+
id=f"https://example.com/entry/{i}",
162
+
title=f"Test Entry {i}",
163
+
link=HttpUrl(f"https://example.com/entry/{i}"),
164
+
updated=datetime.now(),
165
+
)
166
+
store.store_entry("testuser", entry)
167
+
168
+
# List all entries
169
+
entries = store.list_entries("testuser")
170
+
assert len(entries) == 3
171
+
172
+
# List with limit
173
+
limited = store.list_entries("testuser", limit=2)
174
+
assert len(limited) == 2
175
+
176
+
# List for non-existent user
177
+
none_entries = store.list_entries("nonexistent")
178
+
assert len(none_entries) == 0
179
+
180
+
def test_duplicates(self, temp_dir):
181
+
"""Test duplicate management."""
182
+
store = GitStore(temp_dir / "test_repo")
183
+
184
+
# Get initial duplicates (should be empty)
185
+
duplicates = store.get_duplicates()
186
+
assert isinstance(duplicates, DuplicateMap)
187
+
assert len(duplicates.duplicates) == 0
188
+
189
+
# Add duplicate
190
+
store.add_duplicate("https://example.com/dup", "https://example.com/canonical")
191
+
192
+
# Check duplicate was added
193
+
duplicates = store.get_duplicates()
194
+
assert len(duplicates.duplicates) == 1
195
+
assert duplicates.is_duplicate("https://example.com/dup")
196
+
assert duplicates.get_canonical("https://example.com/dup") == "https://example.com/canonical"
197
+
198
+
# Remove duplicate
199
+
result = store.remove_duplicate("https://example.com/dup")
200
+
assert result is True
201
+
202
+
# Check duplicate was removed
203
+
duplicates = store.get_duplicates()
204
+
assert len(duplicates.duplicates) == 0
205
+
assert not duplicates.is_duplicate("https://example.com/dup")
206
+
207
+
def test_search_entries(self, temp_dir):
208
+
"""Test searching entries."""
209
+
store = GitStore(temp_dir / "test_repo")
210
+
211
+
# Add user
212
+
store.add_user("testuser")
213
+
214
+
# Add entries with different content
215
+
entries_data = [
216
+
("Test Python Programming", "Learning Python basics"),
217
+
("JavaScript Tutorial", "Advanced JavaScript concepts"),
218
+
("Python Web Development", "Building web apps with Python"),
219
+
]
220
+
221
+
for title, summary in entries_data:
222
+
entry = AtomEntry(
223
+
id=f"https://example.com/entry/{title.lower().replace(' ', '-')}",
224
+
title=title,
225
+
link=HttpUrl(f"https://example.com/entry/{title.lower().replace(' ', '-')}"),
226
+
updated=datetime.now(),
227
+
summary=summary,
228
+
)
229
+
store.store_entry("testuser", entry)
230
+
231
+
# Search for Python entries
232
+
results = store.search_entries("Python")
233
+
assert len(results) == 2
234
+
235
+
# Search for specific user
236
+
results = store.search_entries("Python", username="testuser")
237
+
assert len(results) == 2
238
+
239
+
# Search with limit
240
+
results = store.search_entries("Python", limit=1)
241
+
assert len(results) == 1
242
+
243
+
# Search for non-existent term
244
+
results = store.search_entries("NonExistent")
245
+
assert len(results) == 0
246
+
247
+
def test_get_stats(self, temp_dir):
248
+
"""Test getting repository statistics."""
249
+
store = GitStore(temp_dir / "test_repo")
250
+
251
+
# Get initial stats
252
+
stats = store.get_stats()
253
+
assert stats["total_users"] == 0
254
+
assert stats["total_entries"] == 0
255
+
assert stats["total_duplicates"] == 0
256
+
257
+
# Add user and entries
258
+
store.add_user("testuser")
259
+
for i in range(3):
260
+
entry = AtomEntry(
261
+
id=f"https://example.com/entry/{i}",
262
+
title=f"Test Entry {i}",
263
+
link=HttpUrl(f"https://example.com/entry/{i}"),
264
+
updated=datetime.now(),
265
+
)
266
+
store.store_entry("testuser", entry)
267
+
268
+
# Add duplicate
269
+
store.add_duplicate("https://example.com/dup", "https://example.com/canonical")
270
+
271
+
# Get updated stats
272
+
stats = store.get_stats()
273
+
assert stats["total_users"] == 1
274
+
assert stats["total_entries"] == 3
275
+
assert stats["total_duplicates"] == 1
276
+
assert "last_updated" in stats
277
+
assert "repository_size" in stats
+353
tests/test_models.py
+353
tests/test_models.py
···
1
+
"""Tests for pydantic models."""
2
+
3
+
from datetime import datetime
4
+
from pathlib import Path
5
+
6
+
import pytest
7
+
from pydantic import HttpUrl, ValidationError
8
+
9
+
from thicket.models import (
10
+
AtomEntry,
11
+
DuplicateMap,
12
+
FeedMetadata,
13
+
ThicketConfig,
14
+
UserConfig,
15
+
UserMetadata,
16
+
)
17
+
18
+
19
+
class TestUserConfig:
20
+
"""Test UserConfig model."""
21
+
22
+
def test_valid_user_config(self):
23
+
"""Test creating valid user config."""
24
+
config = UserConfig(
25
+
username="testuser",
26
+
feeds=["https://example.com/feed.xml"],
27
+
email="test@example.com",
28
+
homepage="https://example.com",
29
+
display_name="Test User",
30
+
)
31
+
32
+
assert config.username == "testuser"
33
+
assert len(config.feeds) == 1
34
+
assert config.feeds[0] == HttpUrl("https://example.com/feed.xml")
35
+
assert config.email == "test@example.com"
36
+
assert config.display_name == "Test User"
37
+
38
+
def test_invalid_email(self):
39
+
"""Test validation of invalid email."""
40
+
with pytest.raises(ValidationError):
41
+
UserConfig(
42
+
username="testuser",
43
+
feeds=["https://example.com/feed.xml"],
44
+
email="invalid-email",
45
+
)
46
+
47
+
def test_invalid_feed_url(self):
48
+
"""Test validation of invalid feed URL."""
49
+
with pytest.raises(ValidationError):
50
+
UserConfig(
51
+
username="testuser",
52
+
feeds=["not-a-url"],
53
+
)
54
+
55
+
def test_optional_fields(self):
56
+
"""Test optional fields with None values."""
57
+
config = UserConfig(
58
+
username="testuser",
59
+
feeds=["https://example.com/feed.xml"],
60
+
)
61
+
62
+
assert config.email is None
63
+
assert config.homepage is None
64
+
assert config.icon is None
65
+
assert config.display_name is None
66
+
67
+
68
+
class TestThicketConfig:
69
+
"""Test ThicketConfig model."""
70
+
71
+
def test_valid_config(self, temp_dir):
72
+
"""Test creating valid configuration."""
73
+
config = ThicketConfig(
74
+
git_store=temp_dir / "git_store",
75
+
cache_dir=temp_dir / "cache",
76
+
users=[
77
+
UserConfig(
78
+
username="testuser",
79
+
feeds=["https://example.com/feed.xml"],
80
+
)
81
+
],
82
+
)
83
+
84
+
assert config.git_store == temp_dir / "git_store"
85
+
assert config.cache_dir == temp_dir / "cache"
86
+
assert len(config.users) == 1
87
+
assert config.users[0].username == "testuser"
88
+
89
+
def test_find_user(self, temp_dir):
90
+
"""Test finding user by username."""
91
+
config = ThicketConfig(
92
+
git_store=temp_dir / "git_store",
93
+
cache_dir=temp_dir / "cache",
94
+
users=[
95
+
UserConfig(username="user1", feeds=["https://example.com/feed1.xml"]),
96
+
UserConfig(username="user2", feeds=["https://example.com/feed2.xml"]),
97
+
],
98
+
)
99
+
100
+
user = config.find_user("user1")
101
+
assert user is not None
102
+
assert user.username == "user1"
103
+
104
+
non_user = config.find_user("nonexistent")
105
+
assert non_user is None
106
+
107
+
def test_add_user(self, temp_dir):
108
+
"""Test adding a new user."""
109
+
config = ThicketConfig(
110
+
git_store=temp_dir / "git_store",
111
+
cache_dir=temp_dir / "cache",
112
+
users=[],
113
+
)
114
+
115
+
new_user = UserConfig(
116
+
username="newuser",
117
+
feeds=["https://example.com/feed.xml"],
118
+
)
119
+
120
+
config.add_user(new_user)
121
+
assert len(config.users) == 1
122
+
assert config.users[0].username == "newuser"
123
+
124
+
def test_add_feed_to_user(self, temp_dir):
125
+
"""Test adding feed to existing user."""
126
+
config = ThicketConfig(
127
+
git_store=temp_dir / "git_store",
128
+
cache_dir=temp_dir / "cache",
129
+
users=[
130
+
UserConfig(username="testuser", feeds=["https://example.com/feed1.xml"]),
131
+
],
132
+
)
133
+
134
+
result = config.add_feed_to_user("testuser", HttpUrl("https://example.com/feed2.xml"))
135
+
assert result is True
136
+
137
+
user = config.find_user("testuser")
138
+
assert len(user.feeds) == 2
139
+
assert HttpUrl("https://example.com/feed2.xml") in user.feeds
140
+
141
+
# Test adding to non-existent user
142
+
result = config.add_feed_to_user("nonexistent", HttpUrl("https://example.com/feed.xml"))
143
+
assert result is False
144
+
145
+
146
+
class TestAtomEntry:
147
+
"""Test AtomEntry model."""
148
+
149
+
def test_valid_entry(self):
150
+
"""Test creating valid Atom entry."""
151
+
entry = AtomEntry(
152
+
id="https://example.com/entry/1",
153
+
title="Test Entry",
154
+
link=HttpUrl("https://example.com/entry/1"),
155
+
updated=datetime.now(),
156
+
published=datetime.now(),
157
+
summary="Test summary",
158
+
content="<p>Test content</p>",
159
+
content_type="html",
160
+
author={"name": "Test Author"},
161
+
categories=["test", "example"],
162
+
)
163
+
164
+
assert entry.id == "https://example.com/entry/1"
165
+
assert entry.title == "Test Entry"
166
+
assert entry.summary == "Test summary"
167
+
assert entry.content == "<p>Test content</p>"
168
+
assert entry.content_type == "html"
169
+
assert entry.author["name"] == "Test Author"
170
+
assert "test" in entry.categories
171
+
172
+
def test_minimal_entry(self):
173
+
"""Test creating minimal Atom entry."""
174
+
entry = AtomEntry(
175
+
id="https://example.com/entry/1",
176
+
title="Test Entry",
177
+
link=HttpUrl("https://example.com/entry/1"),
178
+
updated=datetime.now(),
179
+
)
180
+
181
+
assert entry.id == "https://example.com/entry/1"
182
+
assert entry.title == "Test Entry"
183
+
assert entry.published is None
184
+
assert entry.summary is None
185
+
assert entry.content is None
186
+
assert entry.content_type == "html" # default
187
+
assert entry.author is None
188
+
assert entry.categories == []
189
+
190
+
191
+
class TestDuplicateMap:
192
+
"""Test DuplicateMap model."""
193
+
194
+
def test_empty_duplicates(self):
195
+
"""Test empty duplicate map."""
196
+
dup_map = DuplicateMap()
197
+
assert len(dup_map.duplicates) == 0
198
+
assert not dup_map.is_duplicate("test")
199
+
assert dup_map.get_canonical("test") == "test"
200
+
201
+
def test_add_duplicate(self):
202
+
"""Test adding duplicate mapping."""
203
+
dup_map = DuplicateMap()
204
+
dup_map.add_duplicate("dup1", "canonical1")
205
+
206
+
assert len(dup_map.duplicates) == 1
207
+
assert dup_map.is_duplicate("dup1")
208
+
assert dup_map.get_canonical("dup1") == "canonical1"
209
+
assert dup_map.get_canonical("canonical1") == "canonical1"
210
+
211
+
def test_remove_duplicate(self):
212
+
"""Test removing duplicate mapping."""
213
+
dup_map = DuplicateMap()
214
+
dup_map.add_duplicate("dup1", "canonical1")
215
+
216
+
result = dup_map.remove_duplicate("dup1")
217
+
assert result is True
218
+
assert len(dup_map.duplicates) == 0
219
+
assert not dup_map.is_duplicate("dup1")
220
+
221
+
# Test removing non-existent duplicate
222
+
result = dup_map.remove_duplicate("nonexistent")
223
+
assert result is False
224
+
225
+
def test_get_duplicates_for_canonical(self):
226
+
"""Test getting all duplicates for a canonical ID."""
227
+
dup_map = DuplicateMap()
228
+
dup_map.add_duplicate("dup1", "canonical1")
229
+
dup_map.add_duplicate("dup2", "canonical1")
230
+
dup_map.add_duplicate("dup3", "canonical2")
231
+
232
+
dups = dup_map.get_duplicates_for_canonical("canonical1")
233
+
assert len(dups) == 2
234
+
assert "dup1" in dups
235
+
assert "dup2" in dups
236
+
237
+
dups = dup_map.get_duplicates_for_canonical("canonical2")
238
+
assert len(dups) == 1
239
+
assert "dup3" in dups
240
+
241
+
dups = dup_map.get_duplicates_for_canonical("nonexistent")
242
+
assert len(dups) == 0
243
+
244
+
245
+
class TestFeedMetadata:
246
+
"""Test FeedMetadata model."""
247
+
248
+
def test_valid_metadata(self):
249
+
"""Test creating valid feed metadata."""
250
+
metadata = FeedMetadata(
251
+
title="Test Feed",
252
+
author_name="Test Author",
253
+
author_email="author@example.com",
254
+
author_uri=HttpUrl("https://example.com/author"),
255
+
link=HttpUrl("https://example.com"),
256
+
description="Test description",
257
+
)
258
+
259
+
assert metadata.title == "Test Feed"
260
+
assert metadata.author_name == "Test Author"
261
+
assert metadata.author_email == "author@example.com"
262
+
assert metadata.link == HttpUrl("https://example.com")
263
+
264
+
def test_to_user_config(self):
265
+
"""Test converting metadata to user config."""
266
+
metadata = FeedMetadata(
267
+
title="Test Feed",
268
+
author_name="Test Author",
269
+
author_email="author@example.com",
270
+
author_uri=HttpUrl("https://example.com/author"),
271
+
link=HttpUrl("https://example.com"),
272
+
logo=HttpUrl("https://example.com/logo.png"),
273
+
)
274
+
275
+
feed_url = HttpUrl("https://example.com/feed.xml")
276
+
user_config = metadata.to_user_config("testuser", feed_url)
277
+
278
+
assert user_config.username == "testuser"
279
+
assert user_config.feeds == [feed_url]
280
+
assert user_config.display_name == "Test Author"
281
+
assert user_config.email == "author@example.com"
282
+
assert user_config.homepage == HttpUrl("https://example.com/author")
283
+
assert user_config.icon == HttpUrl("https://example.com/logo.png")
284
+
285
+
def test_to_user_config_fallbacks(self):
286
+
"""Test fallback logic in to_user_config."""
287
+
metadata = FeedMetadata(
288
+
title="Test Feed",
289
+
link=HttpUrl("https://example.com"),
290
+
icon=HttpUrl("https://example.com/icon.png"),
291
+
)
292
+
293
+
feed_url = HttpUrl("https://example.com/feed.xml")
294
+
user_config = metadata.to_user_config("testuser", feed_url)
295
+
296
+
assert user_config.display_name == "Test Feed" # Falls back to title
297
+
assert user_config.homepage == HttpUrl("https://example.com") # Falls back to link
298
+
assert user_config.icon == HttpUrl("https://example.com/icon.png")
299
+
assert user_config.email is None
300
+
301
+
302
+
class TestUserMetadata:
303
+
"""Test UserMetadata model."""
304
+
305
+
def test_valid_metadata(self):
306
+
"""Test creating valid user metadata."""
307
+
now = datetime.now()
308
+
metadata = UserMetadata(
309
+
username="testuser",
310
+
directory="testuser",
311
+
created=now,
312
+
last_updated=now,
313
+
feeds=["https://example.com/feed.xml"],
314
+
entry_count=5,
315
+
)
316
+
317
+
assert metadata.username == "testuser"
318
+
assert metadata.directory == "testuser"
319
+
assert metadata.entry_count == 5
320
+
assert len(metadata.feeds) == 1
321
+
322
+
def test_update_timestamp(self):
323
+
"""Test updating timestamp."""
324
+
now = datetime.now()
325
+
metadata = UserMetadata(
326
+
username="testuser",
327
+
directory="testuser",
328
+
created=now,
329
+
last_updated=now,
330
+
)
331
+
332
+
original_time = metadata.last_updated
333
+
metadata.update_timestamp()
334
+
335
+
assert metadata.last_updated > original_time
336
+
337
+
def test_increment_entry_count(self):
338
+
"""Test incrementing entry count."""
339
+
metadata = UserMetadata(
340
+
username="testuser",
341
+
directory="testuser",
342
+
created=datetime.now(),
343
+
last_updated=datetime.now(),
344
+
entry_count=5,
345
+
)
346
+
347
+
original_count = metadata.entry_count
348
+
original_time = metadata.last_updated
349
+
350
+
metadata.increment_entry_count(3)
351
+
352
+
assert metadata.entry_count == original_count + 3
353
+
assert metadata.last_updated > original_time