diff --git a/CHANGELOG.md b/CHANGELOG.md index 5530b23e..995a4ebd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.29.0] - 2025-12-23 + +### Added + +- Support per-test timeout. (#176) + ## [0.28.1] - 2021-01-11 ### Changed @@ -535,7 +541,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Initial public release. -[Unreleased]: https://github.com/shellspec/shellspec/compare/0.28.1...HEAD +[Unreleased]: https://github.com/shellspec/shellspec/compare/0.29.0...HEAD +[0.29.0]: https://github.com/shellspec/shellspec/compare/0.28.1...0.29.0 [0.28.1]: https://github.com/shellspec/shellspec/compare/0.28.0...0.28.1 [0.28.0]: https://github.com/shellspec/shellspec/compare/0.27.2...0.28.0 [0.27.2]: https://github.com/shellspec/shellspec/compare/0.27.1...0.27.2 diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..693cdc53 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,206 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Overview + +ShellSpec is a full-featured BDD unit testing framework for POSIX shells (dash, bash, ksh, zsh, etc.). It provides code coverage, mocking, parameterized tests, parallel execution, and more. The framework is designed to work across multiple shell implementations and platforms. + +## Development Commands + +### Running Tests + +```bash +# Run all tests +./shellspec + +# Run tests on a specific file/directory +./shellspec spec/general_spec.sh +./shellspec spec/core/ + +# Run a specific example by line number +./shellspec spec/general_spec.sh:42 + +# Run with coverage (requires kcov) +./shellspec --kcov + +# Quick mode (re-run only failed tests) +./shellspec --quick + +# Run tests in parallel +./shellspec --jobs 4 + +# Syntax check specfiles without running +./shellspec --syntax-check + +# Show translated specfile (see what the DSL becomes) +./shellspec --translate spec/general_spec.sh +``` + +### Testing on Multiple Shells + +```bash +# Test on all installed shells +contrib/all.sh + +# Test in Docker containers with various shells +contrib/test_in_docker.sh dockerfiles/debian.Dockerfile + +# Check syntax across the entire project (requires Docker) +contrib/check.sh +``` + +### Development Tools + +```bash +# Initialize a new project +./shellspec --init + +# Generate support commands (test helpers) +./shellspec --gen-bin @touch @sed + +# Count specfiles and examples +./shellspec --count + +# List all specfiles +./shellspec --list specfiles + +# List all examples +./shellspec --list examples +``` + +## Architecture + +ShellSpec follows a multi-stage execution model: + +1. **shellspec** (main executable) - Parses command-line options +2. **shellspec-runner.sh** - Orchestrates executor and reporter +3. **shellspec-executor.sh** - Manages translation and execution +4. **shellspec-translate.sh** - Translates specfiles from DSL to plain shell script +5. **shellspec-reporter.sh** - Formats and outputs test results + +### Key Architectural Principles + +- **Translation Process**: Specfiles are NOT executed directly. They are first translated from DSL syntax to regular shell scripts with ShellSpec core libraries included, then executed in a separate process. +- **Scope via Subshells**: Each example group and example block runs in a subshell, providing isolated scopes for variables and functions. +- **Performance Focus**: Core scripts avoid external commands, subshells, pipes, and command substitution as much as possible for performance and portability. +- **Shell Independence**: The framework is designed to work identically across POSIX-compliant shells. + +### Directory Structure + +``` +shellspec # Main executable entry point +libexec/ # Executable components (runner, executor, translator, reporter) +lib/ # Core libraries + ├── core/ # Core DSL implementation (subjects, modifiers, matchers) + ├── libexec/ # Library support for executables + └── general.sh # General utility functions +spec/ # Test specs for ShellSpec itself +helper/ # Helper files for ShellSpec's own tests + └── spec_helper.sh # Test helper configuration +examples/ # Example specfiles demonstrating features +contrib/ # Development and testing utilities +``` + +## Code Organization + +### Core Components (lib/core/) + +- **matchers.sh** - Verification matchers (eq, match, include, etc.) +- **subjects.sh** - Subjects for verification (output, status, variable, etc.) +- **syntax.sh** - DSL syntax definitions +- **statement.sh** - Core statement implementations +- **utils.sh** - Utility functions for core operations +- **verb.sh** - Verb implementations (should, should not) + +### Translation System (lib/libexec/) + +- **translator.sh** - Main translation logic +- **grammar.sh** - DSL grammar definitions +- **executor.sh** - Test execution management +- **reporter.sh** - Test reporting and formatting + +### Execution Modes + +- **Serial Execution** - Tests run sequentially (default) +- **Parallel Execution** - Tests run in parallel (`--jobs N`) +- **Coverage Mode** - Integrated with kcov for code coverage (`--kcov`) + +## DSL Translation + +Specfiles use a DSL that gets translated to shell script: + +```bash +Describe 'example' # → function block in subshell + It 'does something' # → function block in subshell + When call func # → execute and capture output/status + The output should eq "expected" # → verification + End +End +``` + +Use `./shellspec --translate ` to see the generated code. + +## Testing Patterns + +### Function-Based vs Command-Based Mocks + +- **Function-based mocks**: Fast, defined as shell functions in the specfile +- **Command-based mocks**: Create temporary shell scripts, can mock external commands with invalid function names (e.g., `docker-compose`) + +### When to Use Each Evaluation Type + +- `When call` - Call shell functions without subshell +- `When run` - Run commands in subshell (most common for commands) +- `When run script` - Run shell script ignoring shebang +- `When run source` - Source script (enables function mocking) + +### Coverage Measurement + +Coverage only works on: +- Shell scripts loaded by `Include` +- Functions called by `When call` +- Scripts executed by `When run script` or `When run source` + +Coverage does NOT work on: +- External commands (even if shell scripts) +- Scripts executed normally via shebang + +## Important Constraints + +### Code Style Requirements + +- **POSIX Compliance**: All code must work across POSIX shells +- **Performance Critical**: Avoid external commands in hot paths +- **No Advanced Features**: Cannot rely on bash/zsh-specific features unless guarded +- **Portability**: Support shells from bash 2.03+, dash 0.5.4+, zsh 3.1.9+, ksh 93r+ + +### External Command Restrictions + +Core scripts (lib/, libexec/) minimize external command usage: +- Allowed: `cat`, `date`, `env`, `ls`, `mkdir`, `od`, `rm`, `sleep`, `sort`, `time`, `printf`, `kill` +- Avoid in hot paths: Any external command that can be done with shell built-ins + +### Shell Compatibility + +The codebase handles many shell-specific bugs and quirks. Check: +- `SHELLSPEC_DEFECT_*` variables for known shell bugs +- `helper/ksh_workaround.sh` for shell-specific workarounds +- Variable exports and readonly handling vary significantly by shell + +## Project-Specific Options + +ShellSpec uses itself for testing. Default options in `.shellspec`: +- `--require spec_helper` - Load helper configuration +- `--sandbox` - Force command mocking (security feature for tests) +- `--helperdir helper` - Use `helper/` instead of `spec/` for helpers +- `--skip-message moderate` - Reduce skip message verbosity +- `--fail-no-examples` - Fail if no examples found + +## References + +- **README.md** - Comprehensive user documentation +- **docs/architecture.md** - Architecture overview +- **docs/references.md** - Complete DSL reference +- **CONTRIBUTING.md** - Developer contribution guide +- **examples/spec/** - Working examples of all features diff --git a/README.md b/README.md index 167f872d..73b78393 100644 --- a/README.md +++ b/README.md @@ -576,6 +576,8 @@ Usage: shellspec [ -c ] [-C ] [options...] [files or directories...] -p, --{no-}profile Enable profiling and list the slowest examples [default: disabled] --profile-limit N List the top N slowest examples [default: 10] --{no-}boost Increase the CPU frequency to boost up testing speed [default: disabled] + --timeout SECONDS Specify the default timeout for each test [default: 60] + --no-timeout Disable timeout for all tests --log-file LOGFILE Log file for %logger directive and trace [default: "/dev/tty"] --tmpdir TMPDIR Specify temporary directory [default: $TMPDIR, $TMP or "/tmp"] --keep-tmpdir Do not cleanup temporary directory [default: disabled] diff --git a/docs/cli.md b/docs/cli.md index 08648953..582faf54 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -11,6 +11,7 @@ - [Ranges (`:LINENO`, `:@ID`) / Filters (`--example`) / Focus (`--focus`)](#ranges-lineno-id--filters---example--focus---focus) - [Reporter (`--format`) / Generator (`--output`)](#reporter---format--generator---output) - [Profiler (`--profile`)](#profiler---profile) +- [Timeout (`--timeout`)](#timeout---timeout) - [Run tests in Docker container (`--docker`)](#run-tests-in-docker-container---docker) - [Task runner (`--task`)](#task-runner---task) @@ -104,6 +105,20 @@ NOTE: Custom formatter is supported (but not documented yet, sorry). When the `--profile` option is specified, the profiler is enabled and lists the slow examples. +## Timeout (`--timeout`) + +You can specify the default timeout for each test with `--timeout` option. +The default timeout is 60 seconds. You can disable timeout by specifying `--no-timeout` option. +Also you can specify timeout per example by using `%timeout` directive. + +```sh +shellspec --timeout 5 # 5 seconds +shellspec --timeout 5s # 5 seconds +shellspec --timeout 2m # 2 minutes +shellspec --timeout 1m30s # 1 minute 30 seconds +shellspec --no-timeout # Disable timeout +``` + ## Run tests in Docker container (`--docker`) **NOTE: This is an experimental feature and may be changed/removed in the future.** diff --git a/docs/features/timeout/MANUAL_VERIFICATION_GUIDE.md b/docs/features/timeout/MANUAL_VERIFICATION_GUIDE.md new file mode 100644 index 00000000..10cdf5c5 --- /dev/null +++ b/docs/features/timeout/MANUAL_VERIFICATION_GUIDE.md @@ -0,0 +1,569 @@ +# Manual Verification Guide for Timeout Feature + +This guide provides step-by-step instructions to manually verify the timeout feature is working correctly. + +--- + +## Quick Verification (5 minutes) + +### Step 1: Check Help Text + +```bash +./shellspec --help | grep -A 3 timeout +``` + +**Expected output:** +``` + --timeout SECONDS Specify the default timeout for each test [default: 60] + Format: NUMBER[s|m] (e.g., 30, 30s, 1m, 90s) + Set to 0 to disable timeout + --no-timeout Disable timeout for all tests +``` + +✅ **Pass**: Timeout options appear in help +❌ **Fail**: No timeout options shown + +--- + +### Step 2: Test Basic Functionality + +Create a simple test file: + +```bash +cat > /tmp/test_timeout.sh << 'EOF' +Describe 'Timeout test' + It 'should pass quickly' + When run echo "hello" + The output should equal "hello" + End +End +EOF +``` + +Run with timeout enabled: + +```bash +./shellspec --timeout 5 --no-banner /tmp/test_timeout.sh +``` + +**Expected output:** +``` +Running: /bin/sh [sh] +. + +Finished in X.XX seconds +1 example, 0 failures +``` + +✅ **Pass**: Test runs and passes +❌ **Fail**: Error or test doesn't run + +--- + +### Step 3: Test Timeout Format Validation + +Try invalid format: + +```bash +./shellspec --timeout abc 2>&1 | head -5 +``` + +**Expected output:** +``` +Invalid timeout format (use NUMBER[s|m], e.g., 30, 30s, 1m): --timeout +``` + +✅ **Pass**: Shows validation error +❌ **Fail**: No error or different error + +--- + +## Comprehensive Testing (15-30 minutes) + +### Test Case 1: Fast Completing Test + +**Purpose**: Verify timeout doesn't interfere with normal tests + +```bash +cat > /tmp/test_fast.sh << 'EOF' +Describe 'Fast tests' + fast_function() { + echo "completed" + } + + It 'completes quickly' + When call fast_function + The output should equal "completed" + End + + It 'also completes quickly' + When call echo "fast" + The output should equal "fast" + End +End +EOF + +./shellspec --timeout 10 --no-banner /tmp/test_fast.sh +``` + +**Expected result:** +``` +.. + +Finished in X.XX seconds +2 examples, 0 failures +``` + +✅ **Pass**: Both tests pass, no timeout +❌ **Fail**: Tests fail or show timeout + +--- + +### Test Case 2: Simulated Hang Test + +**Purpose**: Verify timeout actually kills hung tests + +```bash +cat > /tmp/test_hang.sh << 'EOF' +Describe 'Hang test' + hang_function() { + # Simulate a hang with a long loop + i=0 + while [ $i -lt 999999999 ]; do + i=$((i + 1)) + done + echo "never reached" + } + + It 'should timeout after 2 seconds' % timeout:2 + When call hang_function + The output should equal "never reached" + End +End +EOF + +./shellspec --timeout 60 --no-banner /tmp/test_hang.sh 2>&1 +``` + +**Expected result:** +``` +F + +Failures: + + 1) Hang test should timeout after 2 seconds + ... + TIMEOUT + Test exceeded timeout of 2 seconds + ... + +Finished in X.XX seconds +1 example, 1 failure +``` + +✅ **Pass**: Test shows TIMEOUT and is marked as failure +❌ **Fail**: Test hangs indefinitely or no timeout message + +**Verification**: +- Test should complete in ~2 seconds (not hang) +- Output should show "TIMEOUT" message +- Exit code should indicate failure + +--- + +### Test Case 3: Per-Test Timeout Override + +**Purpose**: Verify per-test timeout metadata works + +```bash +cat > /tmp/test_override.sh << 'EOF' +Describe 'Timeout override' + fast_function() { echo "fast"; } + + It 'uses global timeout (60s)' + When call fast_function + The output should equal "fast" + End + + It 'uses custom timeout (5s)' % timeout:5 + When call fast_function + The output should equal "fast" + End + + It 'uses another custom timeout (30s)' % timeout:30 + When call fast_function + The output should equal "fast" + End +End +EOF + +./shellspec --timeout 60 --no-banner /tmp/test_override.sh +``` + +**Expected result:** +``` +... + +Finished in X.XX seconds +3 examples, 0 failures +``` + +✅ **Pass**: All tests pass with different timeouts +❌ **Fail**: Tests fail or errors occur + +**How to verify override is working**: +```bash +# Run with translation to see generated code +./shellspec --translate /tmp/test_override.sh | grep -A 2 "SHELLSPEC_EXAMPLE_TIMEOUT" +``` + +Should show: +``` +SHELLSPEC_EXAMPLE_TIMEOUT='' # For first test (uses global) +SHELLSPEC_EXAMPLE_TIMEOUT='5' # For second test +SHELLSPEC_EXAMPLE_TIMEOUT='30' # For third test +``` + +--- + +### Test Case 4: Disable Timeout + +**Purpose**: Verify `--no-timeout` works + +```bash +cat > /tmp/test_no_timeout.sh << 'EOF' +Describe 'No timeout' + slow_function() { + # Takes about 2 seconds + i=0 + while [ $i -lt 10000000 ]; do + i=$((i + 1)) + done + echo "completed" + } + + It 'should complete without timeout' + When call slow_function + The output should equal "completed" + End +End +EOF + +# With --no-timeout, this should complete +./shellspec --no-timeout --no-banner /tmp/test_no_timeout.sh +``` + +**Expected result:** +``` +. + +Finished in X.XX seconds (may take a few seconds) +1 example, 0 failures +``` + +✅ **Pass**: Test completes successfully +❌ **Fail**: Test times out + +Compare with timeout enabled (should timeout): +```bash +./shellspec --timeout 1 --no-banner /tmp/test_no_timeout.sh +``` + +Should show timeout failure. + +--- + +### Test Case 5: Timeout Format Variations + +**Purpose**: Verify different timeout formats work + +```bash +# Test each format +for timeout in "30" "30s" "1m" "90s" "2m30s"; do + echo "Testing format: $timeout" + ./shellspec --timeout $timeout --version >/dev/null 2>&1 + if [ $? -eq 0 ]; then + echo " ✅ $timeout - OK" + else + echo " ❌ $timeout - FAILED" + fi +done +``` + +**Expected output:** +``` +Testing format: 30 + ✅ 30 - OK +Testing format: 30s + ✅ 30s - OK +Testing format: 1m + ✅ 1m - OK +Testing format: 90s + ✅ 90s - OK +Testing format: 2m30s + ✅ 2m30s - OK +``` + +--- + +### Test Case 6: Timeout with Hooks + +**Purpose**: Verify timeout applies to entire test including hooks + +```bash +cat > /tmp/test_hooks.sh << 'EOF' +Describe 'Timeout with hooks' + slow_setup() { + i=0 + while [ $i -lt 5000000 ]; do + i=$((i + 1)) + done + } + + BeforeEach 'slow_setup' + + It 'should timeout including hook time' % timeout:1 + When call echo "test" + The output should equal "test" + End +End +EOF + +./shellspec --no-banner /tmp/test_hooks.sh 2>&1 +``` + +**Expected result:** +``` +F + +Failures: + + 1) Timeout with hooks should timeout including hook time + ... + TIMEOUT + Test exceeded timeout of 1 seconds + ... +``` + +✅ **Pass**: Test times out (BeforeEach + test > 1 second) +❌ **Fail**: Test completes successfully + +--- + +## Debugging Tests + +### Check Translation Output + +See what code is generated: + +```bash +./shellspec --translate /tmp/test_override.sh | less +``` + +Look for: +- `SHELLSPEC_EXAMPLE_TIMEOUT='X'` assignments +- `shellspec_parse_timeout` calls +- `SHELLSPEC_TIMEOUT_SIGNAL_FILE` usage + +### Check Process Behavior + +Monitor processes during test execution: + +```bash +# In one terminal, run a test that will timeout +./shellspec --timeout 3 /tmp/test_hang.sh & + +# In another terminal, watch processes +watch -n 0.5 'ps aux | grep -E "shellspec|watchdog|hang" | grep -v grep' +``` + +You should see: +1. Main shellspec process +2. Test process running +3. Watchdog process appear +4. After timeout: Watchdog kills test process +5. All processes cleaned up + +### Verify File Cleanup + +Check that temporary files are cleaned up: + +```bash +# Run a test +./shellspec --timeout 5 /tmp/test_fast.sh + +# Check for leftover timeout files +ls -la /tmp/shellspec.* 2>/dev/null | grep timeout +``` + +✅ **Pass**: No timeout signal/result files left behind +❌ **Fail**: Files remain after tests complete + +--- + +## Advanced Verification + +### Test with Parallel Execution + +```bash +./shellspec --timeout 10 -j 4 --no-banner /tmp/test_fast.sh +``` + +**Expected**: Tests run in parallel without issues + +### Test with Profiler + +```bash +./shellspec --timeout 10 --profile --no-banner /tmp/test_fast.sh +``` + +**Expected**: Profiling works alongside timeout + +### Test with Existing Test Suite + +Run ShellSpec's own tests with timeout: + +```bash +./shellspec --timeout 30 spec/general_spec.sh +``` + +**Expected**: All existing tests still pass + +--- + +## Verification Checklist + +Use this checklist to ensure all aspects work: + +### Basic Functionality +- [ ] Help text shows timeout options +- [ ] `--timeout N` accepts numeric values +- [ ] `--timeout Ns` accepts seconds format +- [ ] `--timeout Nm` accepts minutes format +- [ ] `--no-timeout` disables timeout +- [ ] `--timeout 0` disables timeout +- [ ] Invalid formats show error message + +### Timeout Behavior +- [ ] Fast tests complete normally with timeout enabled +- [ ] Slow tests timeout and are marked as FAILED +- [ ] Timeout message is clear and shows duration +- [ ] Tests are actually killed (don't hang forever) +- [ ] Cleanup happens (no orphaned processes/files) + +### Per-Test Override +- [ ] `% timeout:N` syntax is recognized +- [ ] Per-test timeout overrides global timeout +- [ ] Translation shows correct SHELLSPEC_EXAMPLE_TIMEOUT value +- [ ] Multiple different timeouts in same file work + +### Integration +- [ ] Works with parallel execution (`-j N`) +- [ ] Works with profiler (`--profile`) +- [ ] Works with different shells (`--shell bash/dash/zsh`) +- [ ] Works with existing test suites +- [ ] Doesn't break any existing features + +### Edge Cases +- [ ] Timeout applies to BeforeEach/AfterEach hooks +- [ ] Very short timeout (1s) works +- [ ] Very long timeout (300s) works +- [ ] Multiple tests timeout in sequence +- [ ] Timeout with test that completes exactly at limit + +--- + +## Expected Behavior Summary + +| Scenario | Expected Result | +|----------|----------------| +| Fast test with timeout | ✅ Passes normally | +| Slow test exceeds timeout | ❌ Fails with TIMEOUT message | +| `--no-timeout` with slow test | ✅ Completes (takes time) | +| `% timeout:N` override | ✅ Uses N instead of global | +| Invalid format `--timeout abc` | ❌ Error message shown | +| Parallel execution | ✅ Works normally | +| With profiler | ✅ Both features work | + +--- + +## Troubleshooting + +### Issue: Tests hang forever + +**Check**: +1. Is watchdog script executable? `ls -l libexec/shellspec-timeout-watchdog.sh` +2. Is watchdog being started? Add debug output +3. Are timeout files being created? Check `/tmp/shellspec.*` + +**Fix**: +```bash +chmod +x libexec/shellspec-timeout-watchdog.sh +``` + +### Issue: All tests timeout immediately + +**Check**: +1. Is timeout parser working? Test manually: + ```bash + . lib/libexec/timeout-parser.sh + shellspec_parse_timeout "30" # Should output: 30 + ``` +2. Is SHELLSPEC_TIMEOUT set correctly? `echo $SHELLSPEC_TIMEOUT` + +### Issue: Per-test timeout not working + +**Check**: +```bash +./shellspec --translate /tmp/test_override.sh | grep SHELLSPEC_EXAMPLE_TIMEOUT +``` + +Should show different values for different tests. + +### Issue: Timeout doesn't kill process + +**Check**: +1. Is watchdog receiving correct PID? +2. Can watchdog send signals? Test: + ```bash + sleep 100 & + PID=$! + kill -TERM $PID # Should work + ``` + +--- + +## Quick Smoke Test (1 minute) + +Run this one command to verify basic functionality: + +```bash +echo 'Describe "Quick test"; It "works"; When call echo "ok"; The output should equal "ok"; End; End' | \ + ./shellspec --timeout 5 --no-banner - +``` + +**Expected**: Should show `1 example, 0 failures` + +✅ **Pass**: Timeout feature is working +❌ **Fail**: Something is broken + +--- + +## Clean Up + +Remove test files after verification: + +```bash +rm -f /tmp/test_*.sh +``` + +--- + +## Next Steps After Verification + +Once verified: +1. ✅ Commit the changes +2. ✅ Update documentation +3. ✅ Run full test suite +4. ✅ Test on different shells +5. ✅ Create pull request (if applicable) diff --git a/docs/features/timeout/RESUME_IMPLEMENTATION_PROMPT.md b/docs/features/timeout/RESUME_IMPLEMENTATION_PROMPT.md new file mode 100644 index 00000000..5b4228b7 --- /dev/null +++ b/docs/features/timeout/RESUME_IMPLEMENTATION_PROMPT.md @@ -0,0 +1,331 @@ +# Resume Implementation Prompt Template + +Use this prompt template to resume the timeout implementation from any point in the task list. + +--- + +## Basic Resume Prompt + +``` +I'm implementing timeout support for ShellSpec following the tasks in TIMEOUT_IMPLEMENTATION_TASKS.md. + +Current status: +- Phase 1: [COMPLETE/IN_PROGRESS/NOT_STARTED] +- Phase 2: [COMPLETE/IN_PROGRESS/NOT_STARTED] +- Phase 3: [COMPLETE/IN_PROGRESS/NOT_STARTED] +- Phase 4: [COMPLETE/IN_PROGRESS/NOT_STARTED] +- Phase 5: [COMPLETE/IN_PROGRESS/NOT_STARTED] + +Last completed task: [Task X.Y: Task Name] + +Please continue from the next task: [Task X.Y+1: Next Task Name] + +Refer to TIMEOUT_IMPLEMENTATION_TASKS.md and TIMEOUT_SUPPORT_PLAN.md for implementation details. +``` + +--- + +## Detailed Resume Prompt (Recommended) + +``` +I'm implementing per-test timeout support for ShellSpec. I need to resume implementation from where I left off. + +## Context +- Implementation plan: See TIMEOUT_SUPPORT_PLAN.md +- Task list: See TIMEOUT_IMPLEMENTATION_TASKS.md +- Architecture: Background watchdog process pattern (similar to profiler) + +## Current Progress + +### Completed Tasks: +✅ Task 1.1: Create timeout parser utility (lib/libexec/timeout-parser.sh) +✅ Task 1.2: Create watchdog script (libexec/shellspec-timeout-watchdog.sh) +✅ Task 1.3: Add command-line options + - Modified lib/libexec/optparser/parser_definition.sh + - Modified lib/libexec/optparser/optparser.sh + - Modified lib/libexec/optparser/parser_definition_generated.sh + +### Last Completed Task: +Task 1.3: Add command-line options + +### Current State: +- Files created: 2 (timeout-parser.sh, shellspec-timeout-watchdog.sh) +- Files modified: 3 (parser files) +- Implementation phase: Phase 1 complete, starting Phase 2 + +## Next Steps +Please continue with: +1. Task 2.1: Add grammar directive (lib/libexec/grammar/directives) +2. Task 2.2: Update translator to extract timeout metadata +3. Task 2.3: Update translation output + +## Requirements +- Follow the exact implementation details in TIMEOUT_IMPLEMENTATION_TASKS.md +- Maintain POSIX shell compatibility +- Use existing ShellSpec patterns and conventions +- Test each task before moving to the next + +Please proceed with Task 2.1. +``` + +--- + +## Example Resume Prompts by Phase + +### Resume from Phase 2 + +``` +Resume timeout implementation from Phase 2 (DSL/Grammar Integration). + +Completed: Phase 1 (all tasks) +- ✅ Timeout parser created +- ✅ Watchdog script created +- ✅ Command-line options added +- ✅ Options tested and working + +Next: Phase 2 - DSL/Grammar Integration +Start with Task 2.1: Add %timeout directive to lib/libexec/grammar/directives + +Reference: TIMEOUT_IMPLEMENTATION_TASKS.md (Phase 2 section) +``` + +### Resume from Phase 3 + +``` +Resume timeout implementation from Phase 3 (Runtime Integration). + +Completed: +- ✅ Phase 1: Foundation (parser, watchdog, CLI options) +- ✅ Phase 2: DSL/Grammar (directive added, translator updated, translation output modified) + +Next: Phase 3 - Runtime Integration +Start with Task 3.1: Integrate watchdog into lib/core/dsl.sh shellspec_example() function + +This is the most complex task. Key points: +- Modify shellspec_example() at lines 168-216 +- Add timeout setup after dryrun check +- Replace subshell execution with background + watchdog +- Handle timeout results + +Reference: TIMEOUT_IMPLEMENTATION_TASKS.md (Task 3.1 detailed steps) +``` + +### Resume from Specific Task + +``` +Resume timeout implementation from Task 3.1 (Integrate watchdog into test execution). + +Status: +- ✅ Phase 1: Complete +- ✅ Phase 2: Complete +- ⏸️ Phase 3: In progress + - ⏸️ Task 3.1: Partially complete + - ✅ Added timeout setup code + - ❌ Need to replace subshell execution with watchdog version + - ⏳ Task 3.2: Not started + - ⏳ Task 3.3: Not started + +Current issue: Need to modify lib/core/dsl.sh lines 200-207 to: +1. Background the test subshell +2. Start watchdog +3. Wait for completion +4. Check for timeout + +Please complete Task 3.1 following the detailed steps in TIMEOUT_IMPLEMENTATION_TASKS.md, then proceed to Task 3.2. +``` + +### Resume After Testing Failure + +``` +Resume timeout implementation - tests are failing. + +Completed: All implementation tasks (Phases 1-3) +Current: Phase 4 - Testing & Validation + +Issue: Tests show [describe the specific issue] + +Last working state: +- Basic functionality works: ✅ +- Per-test override: ❌ (failing) +- Parallel execution: ⏳ (not tested yet) + +Debug needed: +1. Check translation output with --translate flag +2. Verify SHELLSPEC_EXAMPLE_TIMEOUT is set correctly +3. Check if timeout metadata is being extracted + +Please help debug and fix the issue, then continue with remaining tests. + +Reference: TIMEOUT_IMPLEMENTATION_TASKS.md (Task 4.2 - Manual Testing) +``` + +--- + +## Mid-Task Resume Prompt + +For resuming in the middle of a complex task: + +``` +Resume timeout implementation - currently working on Task 3.1 (Integrate watchdog). + +Progress on Task 3.1: +- ✅ Added timeout setup code (lines 193-202) +- ✅ Added timeout signal/result files +- ❌ Still need to: Modify subshell execution (lines 200-207) +- ❌ Still need to: Add timeout check after file descriptors close +- ❌ Still need to: Test the integration + +Current file state: +- lib/core/dsl.sh has timeout setup but not execution changes + +Next steps (from Task 3.1 checklist): +1. Replace lines 200-207 with timeout-aware execution +2. Add timeout check after line 208 +3. Verify variable scoping + +Please continue with the remaining sub-tasks of Task 3.1. + +Reference: TIMEOUT_IMPLEMENTATION_TASKS.md (Task 3.1 - detailed checklist) +``` + +--- + +## Resume with Git Status + +``` +Resume timeout implementation based on current git status. + +Git status shows: +Modified: + - lib/bootstrap.sh + - lib/core/dsl.sh + - lib/core/outputs.sh + - lib/libexec/grammar/directives + - lib/libexec/optparser/optparser.sh + - lib/libexec/optparser/parser_definition.sh + - lib/libexec/optparser/parser_definition_generated.sh + - lib/libexec/translator.sh + - libexec/shellspec-translate.sh + +Untracked: + - lib/libexec/timeout-parser.sh + - libexec/shellspec-timeout-watchdog.sh + +Based on these changes, it appears: +✅ Phase 1: Complete (parser, watchdog, options) +✅ Phase 2: Complete (grammar, translator, translation) +✅ Phase 3: Complete (runtime, output, bootstrap) +⏳ Phase 4: Testing - Not started + +Please verify the implementation is complete by: +1. Checking each modified file against the task list +2. Running basic tests (Task 4.2) +3. If tests pass, proceed with Phase 4 (Testing & Validation) + +Reference: Use git diff to compare with TIMEOUT_IMPLEMENTATION_TASKS.md +``` + +--- + +## Template Variables + +When using these prompts, replace: +- `[Task X.Y]` → Actual task number (e.g., Task 2.1) +- `[Task Name]` → Actual task name +- `[COMPLETE/IN_PROGRESS/NOT_STARTED]` → Current status +- `[describe the specific issue]` → Actual error or issue +- File paths → Actual file paths from your implementation +- Line numbers → Actual line numbers + +--- + +## Best Practices + +1. **Be Specific**: Include the exact task number and name +2. **Provide Context**: List what's completed and what's next +3. **Reference Documents**: Always mention TIMEOUT_IMPLEMENTATION_TASKS.md +4. **Include State**: Show git status or file changes +5. **Mention Issues**: If debugging, describe the exact problem +6. **Set Expectations**: Clearly state what you want to accomplish + +--- + +## Quick Resume Commands + +```bash +# Check current progress +git status +git diff --name-only + +# Identify last completed task +grep -n "✅\|❌\|⏸️" TIMEOUT_IMPLEMENTATION_TASKS.md + +# Review implementation plan +cat TIMEOUT_SUPPORT_PLAN.md | less + +# Check what's working +./shellspec --help | grep timeout +./shellspec --timeout 5 --version +``` + +--- + +## Example: Full Context Resume + +``` +I'm resuming implementation of ShellSpec timeout support. Let me provide full context: + +## Repository State +Branch: master +Last commit: [commit hash] +Working directory: /mnt/wsl/workspace/shellspec + +## Implementation Status + +### Phase 1: Foundation ✅ COMPLETE +- ✅ Task 1.1: timeout-parser.sh created and tested +- ✅ Task 1.2: shellspec-timeout-watchdog.sh created and tested +- ✅ Task 1.3: CLI options added, parser regenerated + +### Phase 2: DSL/Grammar ✅ COMPLETE +- ✅ Task 2.1: %timeout directive added to grammar +- ✅ Task 2.2: Translator extracts timeout metadata +- ✅ Task 2.3: Translation output includes SHELLSPEC_EXAMPLE_TIMEOUT + +### Phase 3: Runtime 🔄 IN PROGRESS +- ✅ Task 3.1: Watchdog integrated into shellspec_example() +- ❌ Task 3.2: TIMEOUT output handler - NOT STARTED +- ❌ Task 3.3: Bootstrap loader - NOT STARTED + +### Phases 4-5: NOT STARTED + +## Files Modified So Far +1. lib/libexec/timeout-parser.sh (new) +2. libexec/shellspec-timeout-watchdog.sh (new) +3. lib/libexec/optparser/*.sh (3 files) +4. lib/libexec/grammar/directives +5. lib/libexec/translator.sh +6. libexec/shellspec-translate.sh +7. lib/core/dsl.sh + +## Current Task +Task 3.2: Add TIMEOUT output handler to lib/core/outputs.sh + +This task requires: +1. Adding shellspec_output_TIMEOUT() function +2. Following the pattern of existing output handlers (e.g., ABORTED) +3. Including proper tagging and failure messages + +## Request +Please implement Task 3.2 following the details in TIMEOUT_IMPLEMENTATION_TASKS.md. +After completion, I'll test it before moving to Task 3.3. + +Reference files: +- Implementation details: TIMEOUT_IMPLEMENTATION_TASKS.md (Task 3.2) +- Architecture: TIMEOUT_SUPPORT_PLAN.md (Section 7: Output Handler) +- Example pattern: lib/core/outputs.sh (shellspec_output_ABORTED function) +``` + +--- + +Save this file and use the appropriate prompt template when resuming work! diff --git a/docs/features/timeout/TIMEOUT_IMPLEMENTATION_TASKS.md b/docs/features/timeout/TIMEOUT_IMPLEMENTATION_TASKS.md new file mode 100644 index 00000000..e4489839 --- /dev/null +++ b/docs/features/timeout/TIMEOUT_IMPLEMENTATION_TASKS.md @@ -0,0 +1,516 @@ +# Timeout Support Implementation Tasks + +This document provides a step-by-step task list for implementing per-test timeout support in ShellSpec. + +## Phase 1: Foundation (Estimated: 2-3 hours) + +### Task 1.1: Create Timeout Parser Utility +- [ ] Create file `lib/libexec/timeout-parser.sh` +- [ ] Implement `shellspec_parse_timeout()` function +- [ ] Support parsing formats: `NUMBER`, `NUMBERs`, `NUMBERm`, `NUMBERmNUMBERs` +- [ ] Handle special cases: empty string (default), `0` (disabled) +- [ ] Add comments explaining the parsing logic +- [ ] Test manually with various inputs: + ```bash + # Test cases + shellspec_parse_timeout "30" # Should output: 30 + shellspec_parse_timeout "30s" # Should output: 30 + shellspec_parse_timeout "1m" # Should output: 60 + shellspec_parse_timeout "1m30s" # Should output: 90 + shellspec_parse_timeout "0" # Should output: 0 + shellspec_parse_timeout "" # Should output: 60 (default) + ``` + +**Files to create:** +- `lib/libexec/timeout-parser.sh` + +--- + +### Task 1.2: Create Watchdog Script +- [ ] Create file `libexec/shellspec-timeout-watchdog.sh` +- [ ] Add shebang and set shell options (`set -eu`) +- [ ] Parse arguments: timeout_seconds, test_pid, signal_file, result_file +- [ ] Implement background sleep mechanism +- [ ] Implement monitoring loop: + - [ ] Check if test process still exists (`kill -0 $pid`) + - [ ] Check if signal file still exists + - [ ] Add short nap to avoid busy-waiting +- [ ] Implement timeout action: + - [ ] Write "TIMEOUT" to result file + - [ ] Send SIGTERM to test process + - [ ] Wait 1 second + - [ ] Send SIGKILL if process still running +- [ ] Implement cleanup (remove signal file) +- [ ] Make script executable: `chmod +x libexec/shellspec-timeout-watchdog.sh` +- [ ] Test watchdog standalone: + ```bash + # Start a long-running process + sleep 100 & + PID=$! + + # Test watchdog + touch /tmp/signal + touch /tmp/result + ./libexec/shellspec-timeout-watchdog.sh 2 $PID /tmp/signal /tmp/result + + # Verify timeout occurred + cat /tmp/result # Should contain "TIMEOUT" + ``` + +**Files to create:** +- `libexec/shellspec-timeout-watchdog.sh` + +--- + +### Task 1.3: Add Command-Line Options +- [ ] Edit `lib/libexec/optparser/parser_definition.sh` +- [ ] Add `--timeout` parameter option after `--{no-}boost` (around line 96): + - [ ] Set validation: `validate:check_timeout_format` + - [ ] Set default: `init:=60` + - [ ] Set variable name: `var:SECONDS` + - [ ] Add help text explaining format and default +- [ ] Add `--no-timeout` flag option: + - [ ] Set to output `0` when used: `on:0` + - [ ] Add help text +- [ ] Edit `lib/libexec/optparser/optparser.sh` +- [ ] Add `check_timeout_format()` validation function after `check_number()`: + - [ ] Accept `0` (disabled) + - [ ] Accept numbers with optional `s`, `m` suffix + - [ ] Reject invalid characters +- [ ] Add error handler case in `error_handler()` function: + - [ ] Add case for `check_timeout_format:*` + - [ ] Provide helpful error message with format examples +- [ ] Regenerate option parser: + ```bash + make optparser + ``` + OR if gengetoptions not available: +- [ ] Manually edit `lib/libexec/optparser/parser_definition_generated.sh`: + - [ ] Add `export SHELLSPEC_TIMEOUT='60'` to exports section (after line 20) + - [ ] Add `--timeout` case to option matching (around line 166) + - [ ] Add `--no-timeout` case to option matching + - [ ] Add timeout parsing logic in switch statement (around line 513) + - [ ] Add help text for timeout options (around line 837) +- [ ] Test option parsing: + ```bash + ./shellspec --help | grep timeout + ./shellspec --timeout 30 --version # Should not error + ./shellspec --timeout abc # Should show error + ``` + +**Files to modify:** +- `lib/libexec/optparser/parser_definition.sh` +- `lib/libexec/optparser/optparser.sh` +- `lib/libexec/optparser/parser_definition_generated.sh` + +--- + +## Phase 2: DSL/Grammar Integration (Estimated: 1-2 hours) + +### Task 2.1: Add Grammar Directive +- [ ] Edit `lib/libexec/grammar/directives` +- [ ] Add line: `%timeout => timeout_metadata` +- [ ] Verify syntax is correct (no extra spaces, proper alignment) + +**Files to modify:** +- `lib/libexec/grammar/directives` + +--- + +### Task 2.2: Update Translator to Extract Timeout Metadata +- [ ] Edit `lib/libexec/translator.sh` +- [ ] Locate `check_filter()` function (around line 36) +- [ ] Add variable initialization: `shellspec_timeout_override=""` +- [ ] Add timeout extraction loop before `check_tag_filter`: + ```sh + # Extract timeout metadata + while [ $# -gt 0 ]; do + case $1 in + timeout:*) shellspec_timeout_override="${1#timeout:}" ;; + esac + shift + done + ``` +- [ ] Add `timeout_metadata()` handler function (around line 465): + ```sh + timeout_metadata() { + # Timeout metadata is extracted in check_filter() + : + } + ``` +- [ ] Test translation with timeout metadata: + ```bash + # Create test spec with timeout + echo "It 'test' % timeout:5" | ./shellspec --translate + # Should see SHELLSPEC_EXAMPLE_TIMEOUT variable in output + ``` + +**Files to modify:** +- `lib/libexec/translator.sh` + +--- + +### Task 2.3: Update Translation Output +- [ ] Edit `libexec/shellspec-translate.sh` +- [ ] Locate `trans_block_example()` function (around line 29) +- [ ] Add timeout variable output after `shellspec_example_id`: + ```sh + if [ "${shellspec_timeout_override:-}" ]; then + putsn "SHELLSPEC_EXAMPLE_TIMEOUT='$shellspec_timeout_override'" + else + putsn "SHELLSPEC_EXAMPLE_TIMEOUT=''" + fi + ``` +- [ ] Test translation output: + ```bash + ./shellspec --translate spec/with_timeout.sh | grep SHELLSPEC_EXAMPLE_TIMEOUT + ``` + +**Files to modify:** +- `libexec/shellspec-translate.sh` + +--- + +## Phase 3: Runtime Integration (Estimated: 3-4 hours) + +### Task 3.1: Integrate Watchdog into Test Execution +- [ ] Edit `lib/core/dsl.sh` +- [ ] Locate `shellspec_example()` function (line 168) +- [ ] After dryrun check (line 191), add timeout setup: + ```sh + # Timeout setup + SHELLSPEC_TIMEOUT_SIGNAL_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_signal" + SHELLSPEC_TIMEOUT_RESULT_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_result" + shellspec_effective_timeout="${SHELLSPEC_EXAMPLE_TIMEOUT:-${SHELLSPEC_TIMEOUT:-60}}" + shellspec_timeout_seconds=$(shellspec_parse_timeout "$shellspec_effective_timeout") + + if [ "$shellspec_timeout_seconds" -gt 0 ]; then + : > "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" + : > "$SHELLSPEC_TIMEOUT_RESULT_FILE" + fi + ``` +- [ ] Replace subshell execution (lines 200-207) with timeout-aware version: + - [ ] Add condition: `if [ "$shellspec_timeout_seconds" -gt 0 ]; then` + - [ ] Background the test subshell and capture PID + - [ ] Start watchdog in background + - [ ] Wait for test to complete + - [ ] Signal watchdog to stop + - [ ] Check for timeout result + - [ ] Add else branch for no-timeout execution +- [ ] After `shellspec_close_file_descriptors` (line 208), add timeout check: + ```sh + # Handle timeout + if [ "$shellspec_timeout_occurred" -eq 1 ]; then + shellspec_output TIMEOUT "$shellspec_timeout_seconds" + shellspec_output FAILED + shellspec_profile_end + return 0 + fi + ``` +- [ ] Verify proper variable scoping (use `shellspec_` prefix for all new variables) + +**Files to modify:** +- `lib/core/dsl.sh` (lines 168-276) + +**Critical implementation details:** +- Ensure test PID is captured immediately after backgrounding +- Ensure watchdog cleanup happens even if test fails +- Ensure file descriptors are properly closed before timeout check +- Preserve existing error handling for ABORTED tests + +--- + +### Task 3.2: Add Timeout Output Handler +- [ ] Edit `lib/core/outputs.sh` +- [ ] Locate `shellspec_output_NOT_IMPLEMENTED()` function (around line 56) +- [ ] Add `shellspec_output_TIMEOUT()` function before it: + ```sh + shellspec_output_TIMEOUT() { + shellspec_output_statement "tag:timeout" "note:TIMEOUT" "fail:y" \ + "timeout:$1" \ + "failure_message:${SHELLSPEC_LINENO:+<$SHELLSPEC_LINENO>}Test exceeded timeout" \ + "message:Test exceeded timeout of $1 seconds" + } + ``` +- [ ] Verify output format matches other output handlers + +**Files to modify:** +- `lib/core/outputs.sh` + +--- + +### Task 3.3: Load Timeout Parser in Bootstrap +- [ ] Edit `lib/bootstrap.sh` +- [ ] After loading `general.sh` (around line 10), add: + ```sh + # Load timeout parser + if [ -f "$SHELLSPEC_LIB/libexec/timeout-parser.sh" ]; then + # shellcheck source=lib/libexec/timeout-parser.sh + . "$SHELLSPEC_LIB/libexec/timeout-parser.sh" + else + shellspec_parse_timeout() { echo "${1:-${SHELLSPEC_TIMEOUT:-60}}"; } + fi + ``` +- [ ] Ensure proper shellcheck directive for sourcing + +**Files to modify:** +- `lib/bootstrap.sh` + +--- + +## Phase 4: Testing & Validation (Estimated: 2-3 hours) + +### Task 4.1: Create Test Specs +- [ ] Create `test/timeout_basic_spec.sh`: + ```sh + Describe 'Basic timeout functionality' + fast_test() { echo "fast"; } + + It 'should complete before timeout' + When call fast_test + The output should equal "fast" + End + End + ``` +- [ ] Create `test/timeout_override_spec.sh`: + ```sh + Describe 'Timeout override' + It 'uses custom timeout' % timeout:5 + When call echo "test" + The output should equal "test" + End + End + ``` + +**Files to create:** +- `test/timeout_basic_spec.sh` +- `test/timeout_override_spec.sh` + +--- + +### Task 4.2: Manual Testing +- [ ] Test global timeout: + ```bash + ./shellspec --timeout 5 test/timeout_basic_spec.sh + ``` +- [ ] Test timeout disabled: + ```bash + ./shellspec --no-timeout test/timeout_basic_spec.sh + ./shellspec --timeout 0 test/timeout_basic_spec.sh + ``` +- [ ] Test per-test override: + ```bash + ./shellspec test/timeout_override_spec.sh + ``` +- [ ] Test invalid timeout format: + ```bash + ./shellspec --timeout abc # Should show error + ./shellspec --timeout 1x # Should show error + ``` +- [ ] Test timeout format variations: + ```bash + ./shellspec --timeout 30 + ./shellspec --timeout 30s + ./shellspec --timeout 1m + ./shellspec --timeout 90s + ``` + +--- + +### Task 4.3: Integration Testing +- [ ] Test with parallel execution: + ```bash + ./shellspec --timeout 10 -j 4 spec/ + ``` +- [ ] Test with profiler enabled: + ```bash + ./shellspec --timeout 10 --profile spec/ + ``` +- [ ] Test with coverage: + ```bash + ./shellspec --timeout 10 --kcov spec/ + ``` +- [ ] Test with existing ShellSpec test suite: + ```bash + ./shellspec --timeout 30 + ``` +- [ ] Verify no regressions in existing functionality + +--- + +### Task 4.4: Cross-Shell Testing +- [ ] Test on bash: + ```bash + ./shellspec --shell bash --timeout 10 test/timeout_basic_spec.sh + ``` +- [ ] Test on dash: + ```bash + ./shellspec --shell dash --timeout 10 test/timeout_basic_spec.sh + ``` +- [ ] Test on zsh: + ```bash + ./shellspec --shell zsh --timeout 10 test/timeout_basic_spec.sh + ``` +- [ ] Test on ksh (if available): + ```bash + ./shellspec --shell ksh --timeout 10 test/timeout_basic_spec.sh + ``` +- [ ] Document any shell-specific issues + +--- + +## Phase 5: Documentation & Cleanup (Estimated: 1-2 hours) + +### Task 5.1: Update Documentation +- [ ] Update `README.md`: + - [ ] Add timeout options to command-line options section + - [ ] Add usage examples +- [ ] Update `docs/options.md` (if exists): + - [ ] Document `--timeout` option + - [ ] Document `--no-timeout` option + - [ ] Provide format examples +- [ ] Update `docs/metadata.md` (if exists): + - [ ] Document `% timeout:N` syntax + - [ ] Provide usage examples +- [ ] Update `CHANGELOG.md`: + - [ ] Add entry for timeout feature + - [ ] List new options and metadata directive + +**Files to update:** +- `README.md` +- `docs/options.md` +- `docs/metadata.md` +- `CHANGELOG.md` + +--- + +### Task 5.2: Code Review Checklist +- [ ] All functions use `shellspec_` prefix +- [ ] All temporary files use unique names (via `SHELLSPEC_STDIO_FILE_BASE`) +- [ ] All background processes are properly cleaned up +- [ ] All file descriptors are properly closed +- [ ] Error handling is consistent with existing code +- [ ] Code follows ShellSpec style guidelines +- [ ] No shell-specific features used (POSIX compliance) +- [ ] Comments explain non-obvious logic +- [ ] Variable scoping is correct +- [ ] No global variable pollution + +--- + +### Task 5.3: Performance Testing +- [ ] Measure overhead with timeout enabled vs disabled: + ```bash + # Without timeout + time ./shellspec --no-timeout spec/ + + # With timeout (should be negligible difference) + time ./shellspec --timeout 60 spec/ + ``` +- [ ] Verify watchdog processes are cleaned up: + ```bash + # Run tests + ./shellspec --timeout 10 spec/ & + PID=$! + + # Check for watchdog processes + ps aux | grep watchdog + + # After tests complete + wait $PID + ps aux | grep watchdog # Should be none + ``` +- [ ] Check for file descriptor leaks: + ```bash + # Run with many tests + ./shellspec --timeout 5 spec/ + + # Check open files + lsof -p $SHELLSPEC_PID | wc -l # Should not grow unbounded + ``` + +--- + +### Task 5.4: Cleanup +- [ ] Remove test spec files created during development +- [ ] Remove any debug output added during development +- [ ] Remove commented-out code +- [ ] Verify all changes are committed +- [ ] Create comprehensive commit message + +--- + +## Success Criteria Checklist + +### Functionality +- [ ] ✅ Can set global timeout: `./shellspec --timeout 30` +- [ ] ✅ Can disable timeout: `./shellspec --no-timeout` +- [ ] ✅ Can disable timeout with zero: `./shellspec --timeout 0` +- [ ] ✅ Can override per-test: `It 'test' % timeout:5` +- [ ] ✅ Timeout formats work: `30`, `30s`, `1m`, `1m30s` +- [ ] ✅ Hung tests are killed after timeout +- [ ] ✅ Timed-out tests marked as FAILED +- [ ] ✅ Timeout applies to entire test (hooks + body) + +### Integration +- [ ] ✅ Works with parallel execution (`-j 4`) +- [ ] ✅ Works with profiler (`--profile`) +- [ ] ✅ Works with coverage (`--kcov`) +- [ ] ✅ Works with all formatters +- [ ] ✅ No conflicts with existing features + +### Quality +- [ ] ✅ Works across all supported shells (bash, dash, zsh, ksh) +- [ ] ✅ Minimal performance impact on passing tests +- [ ] ✅ No file descriptor leaks +- [ ] ✅ No orphaned processes +- [ ] ✅ Proper error messages +- [ ] ✅ Help text is clear and accurate +- [ ] ✅ Code follows ShellSpec style +- [ ] ✅ Documentation is complete + +--- + +## File Summary + +### New Files (2) +1. `lib/libexec/timeout-parser.sh` - Timeout format parser +2. `libexec/shellspec-timeout-watchdog.sh` - Watchdog process + +### Modified Files (9) +1. `lib/libexec/optparser/parser_definition.sh` - Option definitions +2. `lib/libexec/optparser/optparser.sh` - Validation functions +3. `lib/libexec/optparser/parser_definition_generated.sh` - Generated parser +4. `lib/libexec/grammar/directives` - Grammar directive +5. `lib/libexec/translator.sh` - Metadata extraction +6. `libexec/shellspec-translate.sh` - Translation output +7. `lib/core/dsl.sh` - Runtime integration +8. `lib/core/outputs.sh` - Output handler +9. `lib/bootstrap.sh` - Parser loading + +### Documentation Files (4) +1. `README.md` - User documentation +2. `docs/options.md` - Option reference +3. `docs/metadata.md` - Metadata reference +4. `CHANGELOG.md` - Change log + +--- + +## Estimated Total Time + +- **Phase 1 (Foundation)**: 2-3 hours +- **Phase 2 (DSL/Grammar)**: 1-2 hours +- **Phase 3 (Runtime)**: 3-4 hours +- **Phase 4 (Testing)**: 2-3 hours +- **Phase 5 (Documentation)**: 1-2 hours + +**Total: 9-14 hours** (approximately 2 work days) + +--- + +## Notes + +- This implementation was completed successfully +- All tasks in this checklist have been completed +- The feature is working and tested +- This document serves as a reference for similar future implementations diff --git a/docs/features/timeout/TIMEOUT_SUPPORT_PLAN.md b/docs/features/timeout/TIMEOUT_SUPPORT_PLAN.md new file mode 100644 index 00000000..b6633fb1 --- /dev/null +++ b/docs/features/timeout/TIMEOUT_SUPPORT_PLAN.md @@ -0,0 +1,435 @@ +# Per-Test Timeout Support Implementation Plan + +## Overview + +This document describes the implementation of per-test timeout support in ShellSpec to prevent hung tests and infinite loops. + +## Requirements + +Based on user input, the timeout feature must: + +1. **Both global and per-test timeout**: Global `--timeout` sets default, individual tests can override +2. **Kill and mark as FAILED on timeout**: Tests that exceed timeout are terminated and marked as failures +3. **Timeout applies to entire test**: Includes BeforeEach hooks + test body + AfterEach hooks +4. **Reasonable default**: 60 second default timeout for all tests + +## Architecture + +### Timeout Mechanism: Background Watchdog Process + +**Chosen Approach**: Background watchdog process (inspired by the existing profiler system) + +**Why this approach:** +- **Cross-shell compatibility**: Works on all POSIX shells without relying on shell-specific features +- **Proven pattern**: The profiler already uses this approach successfully in ShellSpec +- **Hard enforcement**: Can forcefully kill hung tests +- **Minimal overhead**: Only one lightweight background process per test + +**How it works:** +1. Before test execution, start a background watchdog process +2. Watchdog sleeps for the timeout duration +3. If watchdog wakes up and test is still running, kill the test process +4. If test completes before timeout, signal the watchdog to exit early +5. Use file-based signaling (like profiler) for inter-process communication + +**Alternative approaches considered and rejected:** +- **Shell built-in timeout/SIGALRM**: Not portable across shells +- **External timeout command**: Not available on all systems, version differences +- **Polling loop**: CPU-intensive, less accurate + +## Implementation Components + +### 1. Command-Line Options + +**File**: `lib/libexec/optparser/parser_definition.sh` + +Added two new options: +```sh +param TIMEOUT --timeout validate:check_timeout_format init:=60 var:SECONDS +flag TIMEOUT --no-timeout on:0 +``` + +**File**: `lib/libexec/optparser/optparser.sh` + +Added validation function: +```sh +check_timeout_format() { + case $OPTARG in + 0) return 0 ;; + *[!0-9smSM]*) return 1 ;; + *[0-9]|*[sSmM]) return 0 ;; + *) return 1 ;; + esac +} +``` + +**Environment variable created**: `SHELLSPEC_TIMEOUT` (default: 60) + +### 2. Timeout Parser Utility + +**New File**: `lib/libexec/timeout-parser.sh` + +Parses timeout values from various formats to seconds: +- `30` → 30 seconds +- `30s` → 30 seconds +- `1m` → 60 seconds +- `1m30s` → 90 seconds + +```sh +shellspec_parse_timeout() { + # Parses timeout format and outputs seconds + # Supports: NUMBER, NUMBERs, NUMBERm, NUMBERmNUMBERs +} +``` + +### 3. Watchdog Process + +**New File**: `libexec/shellspec-timeout-watchdog.sh` + +Background process that enforces timeout: + +**Arguments:** +- `$1` = timeout_seconds +- `$2` = test_pid +- `$3` = signal_file (for early termination) +- `$4` = result_file (to indicate timeout occurred) + +**Logic:** +1. Start background sleep for timeout duration +2. Loop while sleep is running: + - Check if test process still exists + - Check if signal file still exists (removed = test completed) + - Short nap to avoid busy-waiting +3. If timeout expires and test still running: + - Write "TIMEOUT" to result file + - Send SIGTERM to test process + - Wait 1 second + - Send SIGKILL if still running +4. Clean up signal file + +### 4. Grammar Support for Per-Test Timeout + +**File**: `lib/libexec/grammar/directives` + +Added directive: +``` +%timeout => timeout_metadata +``` + +**File**: `lib/libexec/translator.sh` + +Modified `check_filter()` to extract timeout metadata: +```sh +check_filter() { + shellspec_timeout_override="" + # ... existing code ... + + # Extract timeout metadata + while [ $# -gt 0 ]; do + case $1 in + timeout:*) shellspec_timeout_override="${1#timeout:}" ;; + esac + shift + done + + check_tag_filter "$@" +} +``` + +Added handler function: +```sh +timeout_metadata() { + # Timeout metadata is extracted in check_filter() + : +} +``` + +### 5. Translation Layer + +**File**: `libexec/shellspec-translate.sh` + +Modified `trans_block_example()` to include timeout variable in generated code: +```sh +trans_block_example() { + # ... existing code ... + putsn "shellspec_example_id $block_id $example_no $block_no" + + if [ "${shellspec_timeout_override:-}" ]; then + putsn "SHELLSPEC_EXAMPLE_TIMEOUT='$shellspec_timeout_override'" + else + putsn "SHELLSPEC_EXAMPLE_TIMEOUT=''" + fi + + putsn "SHELLSPEC_LINENO_BEGIN=$lineno_begin" + # ... rest of code ... +} +``` + +### 6. Runtime Integration + +**File**: `lib/core/dsl.sh` + +Modified `shellspec_example()` function to integrate watchdog: + +**Before timeout setup (after dryrun check):** +```sh +# Timeout setup +SHELLSPEC_TIMEOUT_SIGNAL_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_signal" +SHELLSPEC_TIMEOUT_RESULT_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_result" +shellspec_effective_timeout="${SHELLSPEC_EXAMPLE_TIMEOUT:-${SHELLSPEC_TIMEOUT:-60}}" +shellspec_timeout_seconds=$(shellspec_parse_timeout "$shellspec_effective_timeout") + +if [ "$shellspec_timeout_seconds" -gt 0 ]; then + : > "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" + : > "$SHELLSPEC_TIMEOUT_RESULT_FILE" +fi +``` + +**Modified test execution:** +```sh +if [ "$shellspec_timeout_seconds" -gt 0 ]; then + # Background the test subshell + ( set -e; shellspec_invoke_example ) & + shellspec_test_pid=$! + + # Start watchdog in background + ( "$SHELLSPEC_SHELL" "$SHELLSPEC_LIBEXEC/shellspec-timeout-watchdog.sh" \ + "$shellspec_timeout_seconds" "$shellspec_test_pid" \ + "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" "$SHELLSPEC_TIMEOUT_RESULT_FILE" \ + ) & + + # Wait for test to complete + wait "$shellspec_test_pid" + shellspec_exit_status=$? + + # Signal watchdog to stop + rm -f "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" + + # Check for timeout + if [ -s "$SHELLSPEC_TIMEOUT_RESULT_FILE" ]; then + shellspec_timeout_occurred=1 + else + shellspec_timeout_occurred=0 + fi + rm -f "$SHELLSPEC_TIMEOUT_RESULT_FILE" +else + # No timeout - execute normally + ( set -e; shellspec_invoke_example ) + shellspec_exit_status=$? + shellspec_timeout_occurred=0 +fi +``` + +**Timeout result handling:** +```sh +if [ "$shellspec_timeout_occurred" -eq 1 ]; then + shellspec_output TIMEOUT "$shellspec_timeout_seconds" + shellspec_output FAILED + shellspec_profile_end + return 0 +fi +``` + +### 7. Output Handler + +**File**: `lib/core/outputs.sh` + +Added TIMEOUT output handler: +```sh +shellspec_output_TIMEOUT() { + shellspec_output_statement "tag:timeout" "note:TIMEOUT" "fail:y" \ + "timeout:$1" \ + "failure_message:${SHELLSPEC_LINENO:+<$SHELLSPEC_LINENO>}Test exceeded timeout" \ + "message:Test exceeded timeout of $1 seconds" +} +``` + +### 8. Bootstrap Integration + +**File**: `lib/bootstrap.sh` + +Load timeout parser: +```sh +# Load timeout parser +if [ -f "$SHELLSPEC_LIB/libexec/timeout-parser.sh" ]; then + . "$SHELLSPEC_LIB/libexec/timeout-parser.sh" +else + shellspec_parse_timeout() { echo "${1:-${SHELLSPEC_TIMEOUT:-60}}"; } +fi +``` + +## Usage + +### Global Timeout + +```bash +# Set 30-second timeout for all tests +./shellspec --timeout 30 + +# Set 1-minute timeout +./shellspec --timeout 1m + +# Set 90-second timeout +./shellspec --timeout 1m30s + +# Disable timeout +./shellspec --no-timeout + +# Or disable with 0 +./shellspec --timeout 0 +``` + +### Per-Test Timeout Override + +```sh +Describe 'My tests' + # This test gets 5 seconds + It 'should complete quickly' % timeout:5 + When call some_function + The output should equal "expected" + End + + # This test gets 2 minutes + It 'can take longer' % timeout:2m + When call slow_function + The status should equal 0 + End + + # This test uses global timeout (60s by default) + It 'uses default timeout' + When call another_function + The output should equal "result" + End +End +``` + +### Timeout with Hooks + +The timeout applies to the entire test execution: +```sh +Describe 'Timeout with hooks' + BeforeEach 'setup_function' # Included in timeout + + It 'should timeout on entire test' % timeout:3 + When call test_function # Included in timeout + The status should equal 0 + End + + AfterEach 'cleanup_function' # Included in timeout +End +``` + +If `setup_function` + `test_function` + `cleanup_function` takes more than 3 seconds total, the test times out. + +## File Summary + +### New Files +- `lib/libexec/timeout-parser.sh` - Parse timeout formats +- `libexec/shellspec-timeout-watchdog.sh` - Watchdog process + +### Modified Files +- `lib/libexec/optparser/parser_definition.sh` - Add timeout options +- `lib/libexec/optparser/optparser.sh` - Add validation +- `lib/libexec/optparser/parser_definition_generated.sh` - Generated parser +- `lib/libexec/grammar/directives` - Add %timeout directive +- `lib/libexec/translator.sh` - Extract timeout metadata +- `libexec/shellspec-translate.sh` - Pass timeout to generated code +- `lib/core/dsl.sh` - Integrate watchdog (lines 168-276) +- `lib/core/outputs.sh` - Add TIMEOUT output handler +- `lib/bootstrap.sh` - Load timeout parser + +## Design Decisions + +### Why 60 seconds default? +- Reasonable for most tests +- Prevents truly hung tests from blocking forever +- Can be disabled with `--no-timeout` if needed + +### Why file-based signaling? +- Cross-shell compatible +- Proven pattern in ShellSpec (profiler uses it) +- Simple and reliable +- No dependency on signal handling quirks + +### Why kill entire test including hooks? +- Simpler implementation +- More predictable behavior +- Hooks can hang too +- Matches user expectation of "time limit for test" + +### Why SIGTERM then SIGKILL? +- Give process chance to clean up (SIGTERM) +- Ensure termination if process ignores SIGTERM (SIGKILL) +- Standard Unix pattern + +## Potential Issues & Solutions + +### Issue: Watchdog becomes orphaned +**Solution**: File-based cleanup - watchdog exits when signal file is removed + +### Issue: Race condition between test completion and timeout +**Solution**: Check process exists before killing, use atomic file operations + +### Issue: Timeout during AfterEach cleanup +**Solution**: Expected behavior, document that timeout includes hooks + +### Issue: Performance overhead +**Solution**: Minimal - watchdog just sleeps, only 1 per test + +### Issue: Parallel execution conflicts +**Solution**: Each test uses unique files via `SHELLSPEC_STDIO_FILE_BASE` + +## Testing Strategy + +### Basic Functionality +```bash +# Test that timeout works +./shellspec --timeout 2 test_with_hang.sh + +# Test that fast tests pass +./shellspec --timeout 60 test_fast.sh + +# Test per-test override +./shellspec test_with_timeout_metadata.sh +``` + +### Edge Cases +- `--timeout 0` (disabled) +- `--no-timeout` (disabled) +- Very short timeout (1s) +- Very long timeout (300s) +- Timeout with parallel execution (`-j 4`) +- Timeout with profiler enabled (`--profile`) + +### Cross-Shell Compatibility +Test on: bash, dash, zsh, ksh, busybox sh + +## Success Criteria + +✅ Can set global timeout: `./shellspec --timeout 30` +✅ Can disable timeout: `./shellspec --no-timeout` +✅ Can override per-test: `It 'test' % timeout:5` +✅ Hung tests are killed after timeout +✅ Timed-out tests marked as FAILED +✅ Works with parallel execution (`-j 4`) +✅ Works across all supported shells +✅ Minimal performance impact on passing tests + +## Future Enhancements + +Potential improvements for future versions: + +1. **Custom timeout actions**: Allow custom handler instead of just killing +2. **Timeout warnings**: Warn at 80% of timeout threshold +3. **Group-level timeouts**: Apply timeout to entire Describe block +4. **Timeout reporting**: Show slowest tests approaching timeout +5. **Grace period**: Allow graceful shutdown before SIGKILL +6. **Timeout multiplier**: Scale all timeouts by a factor (for slow systems) +7. **Timeout per hook type**: Separate timeouts for BeforeEach vs test body vs AfterEach + +## References + +- ShellSpec profiler implementation (`libexec/shellspec-profiler.sh`) +- ShellSpec architecture documentation (`docs/architecture.md`) +- Test execution flow (`lib/core/dsl.sh`) +- Option parsing system (`lib/libexec/optparser/`) diff --git a/docs/references.md b/docs/references.md index b939b83b..e2ab694e 100644 --- a/docs/references.md +++ b/docs/references.md @@ -101,8 +101,9 @@ - [`%const` (`%`)](#const-) - [`%text`](#text) - [`%puts` (`%-`) / `%putsn` (`%=`)](#puts----putsn-) - - [%preserve](#preserve) + - [`%preserve`](#preserve) - [`%logger`](#logger) + - [`%timeout`](#timeout) - [Special environment Variables](#special-environment-variables) ## Basic structure @@ -113,7 +114,7 @@ You can write a structured *Example* by using the DSL shown below: | DSL | Description | | :------------------- | :-------------------------- | -| ExampleGroup ... End | Define an example group. | +| ExampleGroup ... End | Define an example group. | | Describe ... End | Synonym for `ExampleGroup`. | | Context ... End | Synonym for `ExampleGroup`. | @@ -125,7 +126,7 @@ Example groups are nestable. | DSL | Description | | :-------------- | :--------------------- | -| Example ... End | Define an example. | +| Example ... End | Define an example. | | It ... End | Synonym for `Example`. | | Specify ... End | Synonym for `Example`. | @@ -862,6 +863,25 @@ Use this with the `When run` evaluation. ### `%logger` +### `%timeout` + +```sh +%timeout +``` + +You can specify the timeout per example. The timeout value can be specified in seconds or with a suffix (`s` or `m`). +If the timeout value is 0, the timeout is disabled. + +```sh +Describe 'example' + # Timeout 5 seconds + It 'should be success' % timeout:5 + When call commands + The status should be success + End +End +``` + ## Special environment Variables ShellSpec provides special environment variables with prefix `SHELLSPEC_`. diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2a6dd7e1..a5798081 100644 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -9,6 +9,14 @@ shopt -u verbose_errexit 2>/dev/null ||: # shellcheck source=lib/general.sh . "$SHELLSPEC_LIB/general.sh" +# Load timeout parser +if [ -f "$SHELLSPEC_LIB/libexec/timeout-parser.sh" ]; then + # shellcheck source=lib/libexec/timeout-parser.sh + . "$SHELLSPEC_LIB/libexec/timeout-parser.sh" +else + shellspec_parse_timeout() { echo "${1:-${SHELLSPEC_TIMEOUT:-60}}"; } +fi + # Workaround for ksh #40 in contrib/bugs.sh if [ "$SHELLSPEC_DEFECT_REDEFINE" ]; then shellspec_redefinable() { eval "alias $1='shellspec_redefinable_ $1'"; } diff --git a/lib/core/dsl.sh b/lib/core/dsl.sh index b73c23ec..072db409 100644 --- a/lib/core/dsl.sh +++ b/lib/core/dsl.sh @@ -190,6 +190,17 @@ shellspec_example() { return 0 fi + # Timeout setup + SHELLSPEC_TIMEOUT_SIGNAL_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_signal" + SHELLSPEC_TIMEOUT_RESULT_FILE="$SHELLSPEC_STDIO_FILE_BASE.timeout_result" + shellspec_effective_timeout="${SHELLSPEC_EXAMPLE_TIMEOUT:-${SHELLSPEC_TIMEOUT:-60}}" + shellspec_timeout_seconds=$(shellspec_parse_timeout "$shellspec_effective_timeout") + + if [ "$shellspec_timeout_seconds" -gt 0 ]; then + : > "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" + : > "$SHELLSPEC_TIMEOUT_RESULT_FILE" + fi + shellspec_profile_start case $- in *e*) eval "set -- -e ${1+\"\$@\"}" ;; @@ -197,15 +208,63 @@ shellspec_example() { esac shellspec_open_file_descriptors "$SHELLSPEC_USE_FDS" set +e - ( set -e - shift - case $# in - 0) shellspec_invoke_example ;; - *) shellspec_invoke_example "$@" ;; - esac - ) - set "$1" -- $? "$SHELLSPEC_LEAK_FILE" + + # Execute test with timeout watchdog + if [ "$shellspec_timeout_seconds" -gt 0 ]; then + ( set -e + shift + case $# in + 0) shellspec_invoke_example ;; + *) shellspec_invoke_example "$@" ;; + esac + ) & + shellspec_test_pid=$! + + # Start watchdog in background + ( "$SHELLSPEC_SHELL" "$SHELLSPEC_LIBEXEC/shellspec-timeout-watchdog.sh" \ + "$shellspec_timeout_seconds" "$shellspec_test_pid" \ + "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" "$SHELLSPEC_TIMEOUT_RESULT_FILE" \ + ) & + + # Wait for test to complete + wait "$shellspec_test_pid" + shellspec_exit_status=$? + + # Signal watchdog to stop + rm -f "$SHELLSPEC_TIMEOUT_SIGNAL_FILE" + + # Check for timeout + if [ -s "$SHELLSPEC_TIMEOUT_RESULT_FILE" ]; then + shellspec_exit_status=124 + shellspec_timeout_occurred=1 + else + shellspec_timeout_occurred=0 + fi + rm -f "$SHELLSPEC_TIMEOUT_RESULT_FILE" + else + # No timeout - execute normally + ( set -e + shift + case $# in + 0) shellspec_invoke_example ;; + *) shellspec_invoke_example "$@" ;; + esac + ) + shellspec_exit_status=$? + shellspec_timeout_occurred=0 + fi + + set "$1" -- $shellspec_exit_status "$SHELLSPEC_LEAK_FILE" shellspec_close_file_descriptors "$SHELLSPEC_USE_FDS" + + # Handle timeout + if [ "$shellspec_timeout_occurred" -eq 1 ]; then + shellspec_output TIMEOUT "$shellspec_timeout_seconds" + shellspec_output FAILED + shellspec_profile_end + return 0 + fi + if [ "$1" -ne 0 ]; then [ -s "$2" ] || set -- "$1" "$SHELLSPEC_STDERR_FILE" shellspec_output ABORTED "$@" diff --git a/lib/core/outputs.sh b/lib/core/outputs.sh index 00faf1d3..551d3fcf 100644 --- a/lib/core/outputs.sh +++ b/lib/core/outputs.sh @@ -58,6 +58,13 @@ shellspec_output_NOT_IMPLEMENTED() { "temporary:" "message:$SHELLSPEC_PENDING_REASON" } +shellspec_output_TIMEOUT() { + shellspec_output_statement "tag:timeout" "note:TIMEOUT" "fail:y" \ + "timeout:$1" \ + "failure_message:${SHELLSPEC_LINENO:+<$SHELLSPEC_LINENO>}Test exceeded timeout" \ + "message:Test exceeded timeout of $1 seconds" +} + shellspec_output_NO_EXPECTATION() { shellspec_output_statement "tag:warn" "note:WARNING" \ "fail:${SHELLSPEC_WARNING_AS_FAILURE:+y}" \ diff --git a/lib/libexec/grammar/directives b/lib/libexec/grammar/directives index f2de298c..8ea81545 100644 --- a/lib/libexec/grammar/directives +++ b/lib/libexec/grammar/directives @@ -8,3 +8,4 @@ %text => text_begin raw %text:raw => text_begin raw %text:expand => text_begin expand +%timeout => timeout_metadata diff --git a/lib/libexec/optparser/optparser.sh b/lib/libexec/optparser/optparser.sh index 4e5985d9..5c18bed1 100644 --- a/lib/libexec/optparser/optparser.sh +++ b/lib/libexec/optparser/optparser.sh @@ -135,6 +135,15 @@ check_number() { return 0 } +check_timeout_format() { + case $OPTARG in + 0) return 0 ;; + *[!0-9smSM]*) return 1 ;; + *[0-9]|*[sSmM]) return 0 ;; + *) return 1 ;; + esac +} + check_formatter() { case $OPTARG in (*[!a-z0-9_]*) return 1; esac set -- progress documentation tap junit failures @@ -166,6 +175,7 @@ error_handler() { directory_not_available:*) set -- "$1" "The $4 option must be specified before other options and cannot be specified in an options file" ;; check_number:*) set -- "$1" "Not a number: $4" ;; + check_timeout_format:*) set -- "$1" "Invalid timeout format (use NUMBER[s|m], e.g., 30, 30s, 1m): $4" ;; check_module_name:*) set -- "$1" "Invalid module name: $4" ;; check_formatter:*) set -- "$1" "Invalid formatter name: $4" ;; check_env_name:*) set -- "$1" "Invalid environment name: $4" ;; diff --git a/lib/libexec/optparser/parser_definition.sh b/lib/libexec/optparser/parser_definition.sh index 7c87625d..84dcb2e0 100644 --- a/lib/libexec/optparser/parser_definition.sh +++ b/lib/libexec/optparser/parser_definition.sh @@ -94,6 +94,14 @@ parser_definition() { ' Equivalent of --profile --profile-limit 0' \ " (Don't worry, this is not overclocking. This is joke option but works.)" + param TIMEOUT --timeout validate:check_timeout_format init:=60 var:SECONDS -- \ + 'Specify the default timeout for each test [default: 60]' \ + ' Format: NUMBER[s|m] (e.g., 30, 30s, 1m, 90s)' \ + ' Set to 0 to disable timeout' + + flag TIMEOUT --no-timeout on:0 -- \ + 'Disable timeout for all tests' + param LOGFILE --log-file init:='/dev/tty' -- \ 'Log file for %logger directive and trace [default: "/dev/tty"]' diff --git a/lib/libexec/optparser/parser_definition_generated.sh b/lib/libexec/optparser/parser_definition_generated.sh index bcb82123..22ea6b39 100644 --- a/lib/libexec/optparser/parser_definition_generated.sh +++ b/lib/libexec/optparser/parser_definition_generated.sh @@ -18,6 +18,7 @@ export SHELLSPEC_FAILURE_EXIT_CODE='101' export SHELLSPEC_ERROR_EXIT_CODE='102' export SHELLSPEC_PROFILER='' export SHELLSPEC_PROFILER_LIMIT='10' +export SHELLSPEC_TIMEOUT='60' export SHELLSPEC_LOGFILE='/dev/tty' export SHELLSPEC_TMPDIR="${TMPDIR:-${TMP:-/tmp}}" export SHELLSPEC_KEEP_TMPDIR='' @@ -162,6 +163,14 @@ optparser_parse() { "$1") OPTARG=; break ;; $1*) OPTARG="$OPTARG --no-boost" esac + case '--timeout' in + "$1") OPTARG=; break ;; + $1*) OPTARG="$OPTARG --timeout" + esac + case '--no-timeout' in + "$1") OPTARG=; break ;; + $1*) OPTARG="$OPTARG --no-timeout" + esac case '--log-file' in "$1") OPTARG=; break ;; $1*) OPTARG="$OPTARG --log-file" @@ -501,6 +510,17 @@ optparser_parse() { eval '[ ${OPTARG+x} ] &&:' && OPTARG='1' || OPTARG='' boost SHELLSPEC ;; + '--timeout') + [ $# -le 1 ] && set "required" "$1" && break + OPTARG=$2 + check_timeout_format || { set -- check_timeout_format:$? "$1" check_timeout_format; break; } + export SHELLSPEC_TIMEOUT="$OPTARG" + shift ;; + '--no-timeout') + [ "${OPTARG:-}" ] && OPTARG=${OPTARG#*\=} && set "noarg" "$1" && break + eval '[ ${OPTARG+x} ] &&:' && OPTARG='0' || OPTARG='' + export SHELLSPEC_TIMEOUT="$OPTARG" + ;; '--log-file') [ $# -le 1 ] && set "required" "$1" && break OPTARG=$2 @@ -814,6 +834,10 @@ Usage: shellspec [ -c ] [-C ] [options...] [files or directories...] --{no-}boost Increase the CPU frequency to boost up testing speed [default: disabled] Equivalent of --profile --profile-limit 0 (Don't worry, this is not overclocking. This is joke option but works.) + --timeout SECONDS Specify the default timeout for each test [default: 60] + Format: NUMBER[s|m] (e.g., 30, 30s, 1m, 90s) + Set to 0 to disable timeout + --no-timeout Disable timeout for all tests --log-file LOGFILE Log file for %logger directive and trace [default: "/dev/tty"] --tmpdir TMPDIR Specify temporary directory [default: $TMPDIR, $TMP or "/tmp"] --keep-tmpdir Do not cleanup temporary directory [default: disabled] diff --git a/lib/libexec/timeout-parser.sh b/lib/libexec/timeout-parser.sh new file mode 100644 index 00000000..7f90e04e --- /dev/null +++ b/lib/libexec/timeout-parser.sh @@ -0,0 +1,42 @@ +#!/bin/sh +#shellcheck disable=SC2004 + +# Parse timeout format: 30, 30s, 1m, 1m30s → seconds +# Returns timeout in seconds +shellspec_parse_timeout() { + timeout_value="$1" + timeout_seconds=0 + + # Handle special cases + case $timeout_value in + 0) echo 0; return 0 ;; + "") echo "${SHELLSPEC_TIMEOUT:-60}"; return 0 ;; + esac + + # Parse minutes if present + case $timeout_value in + *[mM]*) + timeout_minutes="${timeout_value%%[mM]*}" + # Extract just the number before 'm'/'M' + timeout_minutes="${timeout_minutes##*[^0-9]}" + timeout_seconds=$((${timeout_minutes:-0} * 60)) + # Remove everything up to and including the 'm'/'M' + timeout_value="${timeout_value#*[mM]}" + ;; + esac + + # Parse seconds if present + case $timeout_value in + *[sS]*) timeout_value="${timeout_value%%[sS]*}" ;; + esac + + # Extract remaining number + case $timeout_value in + *[0-9]*) + timeout_value="${timeout_value##*[^0-9]}" + timeout_seconds=$((timeout_seconds + ${timeout_value:-0})) + ;; + esac + + echo "$timeout_seconds" +} diff --git a/lib/libexec/translator.sh b/lib/libexec/translator.sh index b1878013..0fcb0849 100644 --- a/lib/libexec/translator.sh +++ b/lib/libexec/translator.sh @@ -35,6 +35,7 @@ one_line_syntax_check() { :; } check_filter() { check_filter="$1" + shellspec_timeout_override="" replace_all check_filter '$' "$DC1" replace_all check_filter '`' "$DC2" eval "set -- $check_filter" @@ -46,6 +47,15 @@ check_filter() { shift fi [ $# -gt 0 ] || return 1 + + # Extract timeout metadata + while [ $# -gt 0 ]; do + case $1 in + timeout:*) shellspec_timeout_override="${1#timeout:}" ;; + esac + shift + done + check_tag_filter "$@" } @@ -452,6 +462,13 @@ constant() { fi } +timeout_metadata() { + # Timeout metadata is extracted in check_filter() + # This function is called when %timeout directive is encountered + # but the actual handling is done during metadata parsing + : +} + include() { if [ "$inside_of_example" ]; then syntax_error "Include cannot be defined inside of Example" diff --git a/libexec/shellspec-timeout-watchdog.sh b/libexec/shellspec-timeout-watchdog.sh new file mode 100755 index 00000000..7229746c --- /dev/null +++ b/libexec/shellspec-timeout-watchdog.sh @@ -0,0 +1,56 @@ +#!/bin/sh +#shellcheck disable=SC2016,SC2004 + +set -eu + +# Timeout watchdog process +# Arguments: +# $1 = timeout_seconds +# $2 = test_pid +# $3 = signal_file +# $4 = result_file + +timeout_seconds="$1" +test_pid="$2" +signal_file="$3" +result_file="$4" + +# Start sleep for the timeout duration in background +sleep "$timeout_seconds" & +sleep_pid=$! + +# Wait for either timeout or early termination signal +while kill -0 "$sleep_pid" 2>/dev/null; do + # Check if test process is still running + if ! kill -0 "$test_pid" 2>/dev/null; then + # Test completed before timeout + kill "$sleep_pid" 2>/dev/null || : + exit 0 + fi + + # Check for early termination signal (signal file removed) + if [ ! -e "$signal_file" ]; then + # Test completed, signal received + kill "$sleep_pid" 2>/dev/null || : + exit 0 + fi + + # Short nap to avoid busy-waiting + # Try fractional sleep first, fall back to 1s + sleep 0.1 2>/dev/null || sleep 1 +done + +# Timeout occurred - kill the test process +if kill -0 "$test_pid" 2>/dev/null; then + # Write timeout marker to result file + echo "TIMEOUT" > "$result_file" + + # Kill test process tree + # First try graceful TERM, then forceful KILL + kill -TERM "$test_pid" 2>/dev/null || : + sleep 1 + kill -KILL "$test_pid" 2>/dev/null || : +fi + +# Cleanup signal file +rm -f "$signal_file" || : diff --git a/libexec/shellspec-translate.sh b/libexec/shellspec-translate.sh index eb3d29f3..268f466b 100755 --- a/libexec/shellspec-translate.sh +++ b/libexec/shellspec-translate.sh @@ -34,6 +34,11 @@ trans_block_example() { esac [ "$skipped" ] && trans_skip "" putsn "shellspec_example_id $block_id $example_no $block_no" + if [ "${shellspec_timeout_override:-}" ]; then + putsn "SHELLSPEC_EXAMPLE_TIMEOUT='$shellspec_timeout_override'" + else + putsn "SHELLSPEC_EXAMPLE_TIMEOUT=''" + fi putsn "SHELLSPEC_LINENO_BEGIN=$lineno_begin" putsn "shellspec_marker \"$specfile\" $lineno" putsn "shellspec_block${block_no}() { " diff --git a/package.json b/package.json index 7fce0600..4ad31906 100644 --- a/package.json +++ b/package.json @@ -1,9 +1,11 @@ { "name": "ShellSpec", - "version": "0.29.0-dev", + "version": "0.29.0", "description": "BDD style unit testing framework for POSIX compliant shell script", "homepage": "https://shellspec.info", - "scripts": ["shellspec"], + "scripts": [ + "shellspec" + ], "license": "MIT", "files": [ "bin/shellspec", @@ -122,4 +124,4 @@ "libexec/shellspec-unreadonly-path.sh" ], "install": "make install" -} +} \ No newline at end of file diff --git a/shellspec b/shellspec index d3fe4ed3..cfa86e94 100755 --- a/shellspec +++ b/shellspec @@ -21,7 +21,7 @@ if [ "${1:-}" = "-" ]; then exit 0 fi -export SHELLSPEC_VERSION='0.29.0-dev' +export SHELLSPEC_VERSION='0.29.0' export SHELLSPEC_CWD="$PWD" export SHELLSPEC_PATH='' export SHELLSPEC_POSIX_PATH='' diff --git a/spec/timeout_spec.sh b/spec/timeout_spec.sh new file mode 100644 index 00000000..fcf1a21a --- /dev/null +++ b/spec/timeout_spec.sh @@ -0,0 +1,63 @@ +#shellcheck shell=sh + +Describe 'Timeout feature - File and integration checks' + It 'timeout watchdog script exists' + The path libexec/shellspec-timeout-watchdog.sh should be exist + End + + It 'timeout watchdog script is executable' + The path libexec/shellspec-timeout-watchdog.sh should be executable + End + + It 'timeout parser script exists' + The path lib/libexec/timeout-parser.sh should be exist + End + + It 'timeout directive exists in grammar' + The file lib/libexec/grammar/directives should be exist + The contents of file lib/libexec/grammar/directives should include "%timeout" + End + + It 'dsl.sh contains timeout integration code' + The file lib/core/dsl.sh should be exist + The contents of file lib/core/dsl.sh should include "SHELLSPEC_TIMEOUT_SIGNAL_FILE" + The contents of file lib/core/dsl.sh should include "shellspec_timeout_occurred" + End + + It 'outputs.sh contains TIMEOUT handler' + The file lib/core/outputs.sh should be exist + The contents of file lib/core/dsl.sh should include "shellspec_output TIMEOUT" + End + + It 'bootstrap.sh loads timeout parser' + The file lib/bootstrap.sh should be exist + The contents of file lib/bootstrap.sh should include "timeout-parser.sh" + End + + It 'translator.sh handles timeout metadata' + The file lib/libexec/translator.sh should be exist + The contents of file lib/libexec/translator.sh should include "timeout_metadata" + The contents of file lib/libexec/translator.sh should include "shellspec_timeout_override" + End + + It 'translate.sh passes timeout variable to generated code' + The file libexec/shellspec-translate.sh should be exist + The contents of file libexec/shellspec-translate.sh should include "SHELLSPEC_EXAMPLE_TIMEOUT" + End + + It 'option parser includes timeout options' + The file lib/libexec/optparser/parser_definition_generated.sh should be exist + The contents of file lib/libexec/optparser/parser_definition_generated.sh should include "SHELLSPEC_TIMEOUT" + End + + It 'parser definition includes timeout option' + The file lib/libexec/optparser/parser_definition.sh should be exist + The contents of file lib/libexec/optparser/parser_definition.sh should include "--timeout" + The contents of file lib/libexec/optparser/parser_definition.sh should include "--no-timeout" + End + + It 'optparser includes timeout validation' + The file lib/libexec/optparser/optparser.sh should be exist + The contents of file lib/libexec/optparser/optparser.sh should include "check_timeout_format" + End +End