antialias/soroban-abacus-flashcards
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add comprehensive test suite with visual regression testing

Browse Source 
Implement pytest-based testing framework with:
- Unit tests for configuration parsing and core generation logic
- Visual regression tests using perceptual image hashing
- Fixtures and test configuration in conftest.py
- Test documentation and usage guide

Visual tests generate reference images and compare new output against them
to catch regressions while allowing for minor anti-aliasing differences.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Thomas Hallock
2025-09-09 17:29:09 -05:00
parent 11306dfb2e
commit 7a2eb309a8
7 changed files with 797 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
pytest.ini Normal file
View File

@@ -0,0 +1,13 @@
[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
-v
--tb=short
--strict-markers
markers =
slow: marks tests as slow (deselect with '-m "not slow"')
integration: marks tests as integration tests
visual: marks tests as visual regression tests

83
tests/README.md Normal file
View File

@@ -0,0 +1,83 @@
# Testing
This directory contains automated tests for the Soroban flashcard generator.
## Test Structure
- `test_config.py` - Configuration loading and parsing tests
- `test_generation.py` - Core generation logic tests
- `test_visual.py` - Visual regression tests using image comparison
- `conftest.py` - Pytest fixtures and configuration
- `references/` - Reference images for visual regression tests
## Running Tests
### Quick Start
```bash
make pytest-fast # Run unit tests (fast)
make pytest-visual # Run visual regression tests
make pytest # Run all tests
make pytest-cov # Run with coverage report
```
### Direct pytest usage
```bash
# All tests
python3 -m pytest tests/ -v
# Skip slow tests
python3 -m pytest tests/ -v -m "not slow"
# Visual tests only
python3 -m pytest tests/test_visual.py -v
# With coverage
python3 -m pytest tests/ -v --cov=src
```
## Visual Testing
The visual tests generate flashcard images and compare them against reference images using perceptual hashing. This catches visual regressions while allowing for minor differences.
### Updating References
When you make intentional visual changes:
```bash
make update-references
```
This regenerates the reference images in `tests/references/`.
### How Visual Tests Work
1. Generate test images (PNG format, small size for speed)
2. Compare against reference images using `imagehash` library
3. Allow small differences (hash distance < 5) for anti-aliasing variations
4. Fail if images differ significantly, indicating a regression
### Test Philosophy
- **Fast unit tests** for logic and configuration
- **Visual regression tests** for output verification
- **Integration tests** marked as `@pytest.mark.slow`
- **Meaningful failures** with clear error messages
- **Easy maintenance** when the app evolves
## Adding Tests
When adding features:
1. Add unit tests in relevant `test_*.py` file
2. Add visual tests if output changes
3. Update references if visual changes are intentional
4. Use appropriate markers (`@pytest.mark.slow`, etc.)
## CI Integration
Tests are designed to run in CI environments:
- Skip tests requiring typst if not installed
- Use smaller images and lower DPI for speed
- Store reference images in version control
- Clear pass/fail criteria

0
tests/__init__.py Normal file
View File

51
tests/conftest.py Normal file
View File

@@ -0,0 +1,51 @@
"""Pytest configuration and fixtures for soroban flashcard tests."""
import pytest
import tempfile
import shutil
from pathlib import Path
# Add src to path for importing
import sys
sys.path.insert(0, str(Path(__file__).parent.parent / 'src'))
@pytest.fixture
def temp_dir():
"""Create a temporary directory for test outputs."""
temp_path = Path(tempfile.mkdtemp())
yield temp_path
shutil.rmtree(temp_path)
@pytest.fixture
def project_root():
"""Get the project root directory."""
return Path(__file__).parent.parent
@pytest.fixture
def sample_config():
"""Basic test configuration."""
return {
'range': '0-9',
'cards_per_page': 6,
'paper_size': 'us-letter',
'orientation': 'portrait',
'bead_shape': 'diamond',
'color_scheme': 'monochrome',
'font_family': 'DejaVu Sans',
'font_size': '48pt',
'scale_factor': 0.9,
'margins': {
'top': '0.5in',
'bottom': '0.5in',
'left': '0.5in',
'right': '0.5in'
},
'gutter': '5mm'
}
@pytest.fixture
def reference_images_dir(project_root):
"""Directory containing reference images for visual tests."""
ref_dir = project_root / 'tests' / 'references'
ref_dir.mkdir(exist_ok=True)
return ref_dir

153
tests/test_config.py Normal file
View File

@@ -0,0 +1,153 @@
"""Tests for configuration loading and parsing."""
import pytest
import json
import yaml
import tempfile
from pathlib import Path
from generate import load_config, parse_range
class TestConfigLoading:
"""Test configuration file loading."""
def test_load_yaml_config(self, temp_dir):
"""Test loading YAML configuration files."""
config_data = {
'range': '0-9',
'cards_per_page': 4,
'bead_shape': 'circle'
}
config_file = temp_dir / 'test.yaml'
with open(config_file, 'w') as f:
yaml.dump(config_data, f)
loaded = load_config(str(config_file))
assert loaded == config_data
def test_load_json_config(self, temp_dir):
"""Test loading JSON configuration files."""
config_data = {
'range': '0-99',
'color_scheme': 'place-value',
'margins': {'top': '1in', 'bottom': '1in'}
}
config_file = temp_dir / 'test.json'
with open(config_file, 'w') as f:
json.dump(config_data, f)
loaded = load_config(str(config_file))
assert loaded == config_data
def test_load_nonexistent_config(self):
"""Test handling of nonexistent config files."""
with pytest.raises(FileNotFoundError):
load_config('nonexistent.yaml')
class TestRangeParsing:
"""Test number range parsing functionality."""
def test_parse_simple_range(self):
"""Test parsing simple ranges like '0-9'."""
result = parse_range('0-9')
expected = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
assert result == expected
def test_parse_range_with_step(self):
"""Test parsing ranges with custom steps."""
result = parse_range('0-10', step=2)
expected = [0, 2, 4, 6, 8, 10]
assert result == expected
result = parse_range('0-20', step=5)
expected = [0, 5, 10, 15, 20]
assert result == expected
def test_parse_comma_list(self):
"""Test parsing comma-separated number lists."""
result = parse_range('1,5,10,25')
expected = [1, 5, 10, 25]
assert result == expected
def test_parse_mixed_list_and_ranges(self):
"""Test parsing mixed comma-separated values with ranges."""
result = parse_range('1,5-7,10')
expected = [1, 5, 6, 7, 10]
assert result == expected
def test_parse_single_number(self):
"""Test parsing single numbers."""
result = parse_range('42')
expected = [42]
assert result == expected
def test_parse_negative_numbers(self):
"""Test handling of negative numbers (edge case)."""
# Note: The current implementation might not handle this correctly
# This test documents the expected behavior
with pytest.raises(ValueError):
parse_range('-5-5')
def test_parse_invalid_range(self):
"""Test handling of invalid range strings."""
with pytest.raises(ValueError):
parse_range('invalid')
with pytest.raises(ValueError):
parse_range('10-5') # End before start
def test_parse_large_range(self):
"""Test parsing large ranges."""
result = parse_range('0-100', step=25)
expected = [0, 25, 50, 75, 100]
assert result == expected
def test_parse_step_ignored_for_lists(self):
"""Test that step parameter is ignored for comma-separated lists."""
result = parse_range('1,2,3,4', step=10) # Step should be ignored
expected = [1, 2, 3, 4]
assert result == expected
class TestConfigIntegration:
"""Integration tests for configuration handling."""
def test_real_config_files(self, project_root):
"""Test loading real configuration files from the project."""
config_dir = project_root / 'config'
# Test default config
default_config = config_dir / 'default.yaml'
if default_config.exists():
config = load_config(str(default_config))
assert 'range' in config
assert 'cards_per_page' in config
def test_config_with_range_parsing(self, temp_dir):
"""Test integration of config loading with range parsing."""
config_data = {
'range': '5-15',
'step': 2,
'cards_per_page': 8
}
config_file = temp_dir / 'test.yaml'
with open(config_file, 'w') as f:
yaml.dump(config_data, f)
config = load_config(str(config_file))
numbers = parse_range(config['range'], config.get('step', 1))
expected = [5, 7, 9, 11, 13, 15]
assert numbers == expected
def test_config_defaults(self, sample_config):
"""Test that sample config has reasonable defaults."""
assert sample_config['cards_per_page'] > 0
assert sample_config['scale_factor'] > 0
assert sample_config['bead_shape'] in ['diamond', 'circle', 'square']
assert sample_config['color_scheme'] in ['monochrome', 'place-value', 'heaven-earth', 'alternating']

260
tests/test_generation.py Normal file
View File

@@ -0,0 +1,260 @@
"""Tests for core generation functionality."""
import pytest
import tempfile
from pathlib import Path
from generate import generate_single_card_typst, generate_typst_file
class TestTypstGeneration:
"""Test Typst file generation functions."""
def test_generate_single_card_typst_front(self, temp_dir, sample_config):
"""Test generation of single card Typst file for front side."""
output_file = temp_dir / 'card_front.typ'
generate_single_card_typst(
number=7,
side='front',
config=sample_config,
output_path=output_file,
project_root=temp_dir
)
assert output_file.exists()
content = output_file.read_text()
assert '#import "single-card.typ"' in content
assert 'generate-single-card' in content
assert '7,' in content # Number should be in the content
assert 'side: "front"' in content
assert 'bead-shape: "diamond"' in content # From sample config
def test_generate_single_card_typst_back(self, temp_dir, sample_config):
"""Test generation of single card Typst file for back side."""
output_file = temp_dir / 'card_back.typ'
generate_single_card_typst(
number=42,
side='back',
config=sample_config,
output_path=output_file,
project_root=temp_dir
)
assert output_file.exists()
content = output_file.read_text()
assert 'side: "back"' in content
assert '42,' in content
def test_generate_single_card_with_custom_config(self, temp_dir):
"""Test single card generation with custom configuration."""
custom_config = {
'bead_shape': 'circle',
'color_scheme': 'place-value',
'colored_numerals': True,
'hide_inactive_beads': True,
'transparent': True,
'card_width': '4in',
'card_height': '3in',
'font_size': '36pt',
'font_family': 'Arial',
'scale_factor': 1.2
}
output_file = temp_dir / 'custom_card.typ'
generate_single_card_typst(
number=123,
side='front',
config=custom_config,
output_path=output_file,
project_root=temp_dir
)
content = output_file.read_text()
assert 'bead-shape: "circle"' in content
assert 'color-scheme: "place-value"' in content
assert 'colored-numerals: true' in content
assert 'hide-inactive-beads: true' in content
assert 'transparent: true' in content
assert 'width: 4in' in content
assert 'height: 3in' in content
assert 'font-size: 36pt' in content
assert 'font-family: "Arial"' in content
assert 'scale-factor: 1.2' in content
def test_generate_typst_file_basic(self, temp_dir, sample_config):
"""Test generation of multi-card Typst file."""
numbers = [1, 2, 3, 5, 8]
output_file = temp_dir / 'flashcards.typ'
generate_typst_file(numbers, sample_config, output_file)
assert output_file.exists()
content = output_file.read_text()
assert '#import "templates/flashcards.typ"' in content
assert 'generate-flashcards' in content
assert '(1, 2, 3, 5, 8,)' in content # Numbers array
assert 'cards-per-page: 6' in content # From sample config
def test_generate_typst_file_with_margins(self, temp_dir):
"""Test Typst generation with custom margins."""
config = {
'cards_per_page': 4,
'paper_size': 'a4',
'orientation': 'landscape',
'margins': {
'top': '1in',
'bottom': '0.75in',
'left': '0.5in',
'right': '0.5in'
},
'gutter': '10mm',
'font_family': 'Times',
'font_size': '36pt'
}
numbers = [10, 20, 30]
output_file = temp_dir / 'custom_flashcards.typ'
generate_typst_file(numbers, config, output_file)
content = output_file.read_text()
assert 'paper-size: "a4"' in content
assert 'orientation: "landscape"' in content
assert 'top: 1in' in content
assert 'bottom: 0.75in' in content
assert 'gutter: 10mm' in content
assert 'font-family: "Times"' in content
def test_generate_typst_file_boolean_options(self, temp_dir):
"""Test Typst generation with boolean configuration options."""
config = {
'cards_per_page': 6,
'show_cut_marks': True,
'show_registration': False,
'show_empty_columns': True,
'hide_inactive_beads': False,
'colored_numerals': True
}
numbers = [7]
output_file = temp_dir / 'boolean_test.typ'
generate_typst_file(numbers, config, output_file)
content = output_file.read_text()
assert 'show-cut-marks: true' in content
assert 'show-registration: false' in content
assert 'show-empty-columns: true' in content
assert 'hide-inactive-beads: false' in content
assert 'colored-numerals: true' in content
def test_generate_typst_file_empty_numbers(self, temp_dir, sample_config):
"""Test handling of empty numbers list."""
numbers = []
output_file = temp_dir / 'empty.typ'
generate_typst_file(numbers, sample_config, output_file)
content = output_file.read_text()
assert '()' in content # Empty array
def test_generate_typst_file_single_number(self, temp_dir, sample_config):
"""Test generation with single number."""
numbers = [99]
output_file = temp_dir / 'single.typ'
generate_typst_file(numbers, sample_config, output_file)
content = output_file.read_text()
assert '(99,)' in content # Single element array with trailing comma
class TestConfigDefaults:
"""Test default value handling in generation functions."""
def test_single_card_defaults(self, temp_dir):
"""Test that single card generation handles missing config gracefully."""
minimal_config = {}
output_file = temp_dir / 'defaults.typ'
generate_single_card_typst(
number=5,
side='front',
config=minimal_config,
output_path=output_file,
project_root=temp_dir
)
content = output_file.read_text()
# Should use defaults from the function
assert 'bead-shape: "diamond"' in content
assert 'color-scheme: "monochrome"' in content
assert 'colored-numerals: false' in content
assert 'hide-inactive-beads: false' in content
assert 'columns: auto' in content
assert 'font-family: "DejaVu Sans"' in content
def test_typst_file_defaults(self, temp_dir):
"""Test that multi-card generation handles missing config gracefully."""
minimal_config = {}
numbers = [1, 2]
output_file = temp_dir / 'defaults.typ'
generate_typst_file(numbers, minimal_config, output_file)
content = output_file.read_text()
# Should use defaults
assert 'cards-per-page: 6' in content
assert 'paper-size: "us-letter"' in content
assert 'orientation: "portrait"' in content
assert 'font-family: "DejaVu Sans"' in content
class TestEdgeCases:
"""Test edge cases and error conditions."""
def test_large_numbers(self, temp_dir, sample_config):
"""Test generation with large numbers."""
large_numbers = [999, 1000, 9999]
output_file = temp_dir / 'large.typ'
generate_typst_file(large_numbers, sample_config, output_file)
content = output_file.read_text()
assert '(999, 1000, 9999,)' in content
def test_zero_handling(self, temp_dir, sample_config):
"""Test generation with zero."""
numbers = [0]
output_file = temp_dir / 'zero.typ'
generate_single_card_typst(
number=0,
side='front',
config=sample_config,
output_path=output_file,
project_root=temp_dir
)
content = output_file.read_text()
assert '0,' in content # Zero should be handled correctly
def test_special_characters_in_paths(self, temp_dir, sample_config):
"""Test generation with paths containing spaces."""
numbers = [5]
# Create directory with spaces
special_dir = temp_dir / 'path with spaces'
special_dir.mkdir()
output_file = special_dir / 'test file.typ'
generate_typst_file(numbers, sample_config, output_file)
assert output_file.exists()
content = output_file.read_text()
assert '(5,)' in content

237
tests/test_visual.py Normal file
View File

@@ -0,0 +1,237 @@
"""Visual regression tests for flashcard generation."""
import pytest
import subprocess
import tempfile
import imagehash
from pathlib import Path
from PIL import Image
import sys
# Import our modules
from generate import generate_cards_direct
class TestVisualRegression:
"""Visual regression tests using image hashing and comparison."""
def test_basic_card_generation(self, temp_dir, sample_config, reference_images_dir):
"""Test basic card generation produces consistent output."""
# Generate a single test card (number 7)
numbers = [7]
config = {
**sample_config,
'transparent': False,
'card_width': '300px', # Smaller for faster tests
'card_height': '200px'
}
# Generate test output
test_output = temp_dir / 'test_cards'
generated_files = generate_cards_direct(
numbers, config, test_output,
format='png', dpi=150, # Lower DPI for faster tests
separate_fronts_backs=True
)
assert len(generated_files) == 2 # Front and back
# Check that files were created
front_file = test_output / 'fronts' / 'card_000.png'
back_file = test_output / 'backs' / 'card_000.png'
assert front_file.exists(), "Front card image should exist"
assert back_file.exists(), "Back card image should exist"
# Basic image validation
front_img = Image.open(front_file)
back_img = Image.open(back_file)
# Check dimensions are reasonable
assert front_img.size[0] > 100, "Front image should have reasonable width"
assert front_img.size[1] > 100, "Front image should have reasonable height"
assert back_img.size[0] > 100, "Back image should have reasonable width"
assert back_img.size[1] > 100, "Back image should have reasonable height"
# Store as reference if they don't exist
ref_front = reference_images_dir / 'card_7_front.png'
ref_back = reference_images_dir / 'card_7_back.png'
if not ref_front.exists():
front_img.save(ref_front)
print(f"Created reference image: {ref_front}")
if not ref_back.exists():
back_img.save(ref_back)
print(f"Created reference image: {ref_back}")
# If references exist, compare hashes
if ref_front.exists() and ref_back.exists():
ref_front_img = Image.open(ref_front)
ref_back_img = Image.open(ref_back)
# Use perceptual hashing for reasonable tolerance
front_hash = imagehash.phash(front_img)
ref_front_hash = imagehash.phash(ref_front_img)
back_hash = imagehash.phash(back_img)
ref_back_hash = imagehash.phash(ref_back_img)
# Allow small differences (hash distance < 5)
assert front_hash - ref_front_hash < 5, f"Front card changed significantly (hash diff: {front_hash - ref_front_hash})"
assert back_hash - ref_back_hash < 5, f"Back card changed significantly (hash diff: {back_hash - ref_back_hash})"
def test_different_bead_shapes(self, temp_dir, sample_config, reference_images_dir):
"""Test that different bead shapes produce visually different outputs."""
numbers = [5] # Simple number for shape testing
base_config = {
**sample_config,
'card_width': '300px',
'card_height': '200px'
}
shapes = ['diamond', 'circle', 'square']
shape_hashes = {}
for shape in shapes:
config = {**base_config, 'bead_shape': shape}
shape_output = temp_dir / f'shape_{shape}'
generated_files = generate_cards_direct(
numbers, config, shape_output,
format='png', dpi=150,
separate_fronts_backs=True
)
assert len(generated_files) == 2
# Get hash of front image (where shape is visible)
front_file = shape_output / 'fronts' / 'card_000.png'
front_img = Image.open(front_file)
shape_hashes[shape] = imagehash.phash(front_img)
# Verify shapes produce different images
assert shape_hashes['diamond'] - shape_hashes['circle'] > 3, "Diamond and circle shapes should be visually different"
assert shape_hashes['diamond'] - shape_hashes['square'] > 3, "Diamond and square shapes should be visually different"
assert shape_hashes['circle'] - shape_hashes['square'] > 3, "Circle and square shapes should be visually different"
def test_color_schemes(self, temp_dir, sample_config, reference_images_dir):
"""Test that different color schemes produce different outputs."""
numbers = [23] # Multi-digit number for color testing
base_config = {
**sample_config,
'card_width': '300px',
'card_height': '200px'
}
schemes = ['monochrome', 'place-value']
scheme_hashes = {}
for scheme in schemes:
config = {**base_config, 'color_scheme': scheme}
scheme_output = temp_dir / f'color_{scheme}'
generated_files = generate_cards_direct(
numbers, config, scheme_output,
format='png', dpi=150,
separate_fronts_backs=True
)
assert len(generated_files) == 2
# Get hash of front image
front_file = scheme_output / 'fronts' / 'card_000.png'
front_img = Image.open(front_file)
scheme_hashes[scheme] = imagehash.phash(front_img)
# Color schemes should produce different images
hash_diff = scheme_hashes['monochrome'] - scheme_hashes['place-value']
assert hash_diff > 2, f"Color schemes should be visually different (hash diff: {hash_diff})"
@pytest.mark.slow
def test_pdf_generation_structure(self, temp_dir, sample_config):
"""Test that PDF generation produces valid PDF files."""
# This test requires typst to be installed
try:
subprocess.run(['typst', '--version'], capture_output=True, check=True)
except (subprocess.CalledProcessError, FileNotFoundError):
pytest.skip("Typst not installed - skipping PDF tests")
from generate import generate_typst_file
numbers = [1, 2, 3]
output_typst = temp_dir / 'test.typ'
output_pdf = temp_dir / 'test.pdf'
# Generate Typst file
generate_typst_file(numbers, sample_config, output_typst)
# Check Typst file was created and has content
assert output_typst.exists()
assert output_typst.stat().st_size > 100 # Should have substantial content
# Try to compile to PDF (if typst is available)
try:
result = subprocess.run([
'typst', 'compile',
'--font-path', str(Path(__file__).parent.parent / 'fonts'),
str(output_typst), str(output_pdf)
], capture_output=True, text=True, cwd=temp_dir)
if result.returncode == 0:
assert output_pdf.exists(), "PDF should be generated"
assert output_pdf.stat().st_size > 1000, "PDF should have reasonable size"
else:
print(f"PDF compilation failed: {result.stderr}")
# Don't fail the test - typst might have font issues in test environment
except FileNotFoundError:
pytest.skip("Typst not available for PDF compilation")
def test_reference_image_update_utility(self, temp_dir, sample_config, reference_images_dir):
"""Utility to regenerate reference images when needed."""
# This test can be run manually to update references
# Skip in normal test runs
if not pytest.config.getoption("--update-references", default=False):
pytest.skip("Reference update not requested")
# Generate fresh reference images
test_cases = [
(7, 'basic'),
(23, 'multidigit'),
(0, 'zero')
]
for number, name in test_cases:
config = {
**sample_config,
'card_width': '300px',
'card_height': '200px',
'transparent': False
}
output_dir = temp_dir / f'ref_{name}'
generate_cards_direct(
[number], config, output_dir,
format='png', dpi=150,
separate_fronts_backs=True
)
# Copy to references
front_src = output_dir / 'fronts' / 'card_000.png'
back_src = output_dir / 'backs' / 'card_000.png'
front_dst = reference_images_dir / f'card_{number}_front.png'
back_dst = reference_images_dir / f'card_{number}_back.png'
if front_src.exists():
front_src.replace(front_dst)
print(f"Updated reference: {front_dst}")
if back_src.exists():
back_src.replace(back_dst)
print(f"Updated reference: {back_dst}")
def pytest_addoption(parser):
"""Add custom pytest options."""
parser.addoption(
"--update-references", action="store_true", default=False,
help="Update reference images for visual tests"
)