Skip to content

Add acceptance tests for workflow, run, and cache commands #9766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
williammartin merged 17 commits into from
Oct 17, 2024
Merged

Add acceptance tests for workflow, run, and cache commands #9766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c3ffdc9
implement base workflow list acceptance test
BagToad Oct 15, 2024
5f919bc
implement workflow enable/disable acceptance test
BagToad Oct 15, 2024
0b1d692
Create workflow-view.txtar
BagToad Oct 15, 2024
3737f10
Create workflow-run.txtar
BagToad Oct 15, 2024
446ffed
Create run-cancel.txtar
BagToad Oct 15, 2024
3f080e0
Create run-delete.txtar
BagToad Oct 15, 2024
3d3e061
Create run-download.txtar
BagToad Oct 15, 2024
745863b
Create run-rerun.txtar
BagToad Oct 15, 2024
682b75f
Create run-view.txtar
BagToad Oct 15, 2024
924d81a
Create cache-list-delete.txtar
BagToad Oct 15, 2024
8d62631
Update run-rerun.txtar
BagToad Oct 15, 2024
4ce8236
sleep 10s before checking for workflow run
BagToad Oct 15, 2024
8f567e2
Watch for run to end for cancel Acceptance test
williammartin Oct 17, 2024
93081c9
Use regex assert for enable disable workflow Acceptance test
williammartin Oct 17, 2024
0f62ec2
Adjust sleeps to echos in Acceptance workflows
williammartin Oct 17, 2024
07ea43a
Remove empty stdout checks
williammartin Oct 17, 2024
dc3e805
Remove unnecesary mkdir from download Acceptance test
williammartin Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions acceptance/acceptance_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ import (
"fmt"
"os"
"path"
"strconv"
"strings"
"testing"
"time"

"math/rand"

Expand Down Expand Up @@ -35,6 +37,16 @@ func TestPullRequests(t *testing.T) {
testscript.Run(t, testScriptParamsFor(tsEnv, "pr"))
}

func TestWorkflows(t *testing.T) {
var tsEnv testScriptEnv
if err := tsEnv.fromEnv(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}

testscript.Run(t, testScriptParamsFor(tsEnv, "workflow"))
}

func testScriptParamsFor(tsEnv testScriptEnv, command string) testscript.Params {
var files []string
if tsEnv.script != "" {
Expand Down Expand Up @@ -119,6 +131,23 @@ func sharedCmds(tsEnv testScriptEnv) map[string]func(ts *testscript.TestScript,

ts.Setenv(args[0], strings.TrimRight(ts.ReadFile("stdout"), "\n"))
},
"sleep": func(ts *testscript.TestScript, neg bool, args []string) {

@williammartin williammartin Oct 17, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While reviewing this, @BagToad and I noodled on some ideas to avoid flakes around sleeping for arbitrary amounts of time. Both approaches required a new command. One would be a eventuallyexec single command which would re-exec a command repeatedly until the exit code was 0. Then we could use jq to hackily cause an error based on some state e.g.:

gh run list --json name,status --jq 'if .[0].name != "EXPECTED" then halt_error else .[0].name end'

A perhaps better approach we spiked in f98f45f

Which allows for retryable stdout assertions like so:

retryableexec gh workflow view 'Test Workflow Name'
eventuallystdout 'Test Workflow Name - workflow.yml'

if neg {
ts.Fatalf("unsupported: ! sleep")
}
if len(args) != 1 {
ts.Fatalf("usage: sleep seconds")
}

// sleep for the given number of seconds
seconds, err := strconv.Atoi(args[0])
if err != nil {
ts.Fatalf("invalid number of seconds: %v", err)
}

d := time.Duration(seconds) * time.Second
time.Sleep(d)
},
}
}

Expand Down
69 changes: 69 additions & 0 deletions acceptance/testdata/workflow/cache-list-delete.txtar
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Use gh as a credential helper
exec gh auth setup-git

# Create a repository with a file so it has a default branch
exec gh repo create $ORG/$SCRIPT_NAME-$RANDOM_STRING --add-readme --private

# Defer repo cleanup
defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING

# Clone the repo
exec gh repo clone $ORG/$SCRIPT_NAME-$RANDOM_STRING

# commit the workflow file
cd $SCRIPT_NAME-$RANDOM_STRING
mkdir .github/workflows
mv ../workflow.yml .github/workflows/workflow.yml
exec git add .github/workflows/workflow.yml
exec git commit -m 'Create workflow file'
exec git push -u origin main

# Sleep because it takes a second for the workflow to register
sleep 1

# Check the workflow is indeed created
exec gh workflow list
stdout 'Test Workflow Name'

# Run the workflow
exec gh workflow run 'Test Workflow Name'

# It takes some time for a workflow run to register
sleep 10

# Get the run ID we want to watch
exec gh run list --json databaseId --jq '.[0].databaseId'
stdout2env RUN_ID

# Wait for workflow to complete
exec gh run watch $RUN_ID --exit-status

# List the cache
exec gh cache list
stdout 'Linux-values'

# Delete the cache
exec gh cache delete 'Linux-values'

-- workflow.yml --
name: Test Workflow Name

on: workflow_dispatch

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Cache values
id: cache-values
uses: actions/cache@v4
with:
path: values.txt
key: ${{ runner.os }}-values

- name: Generate values file
if: steps.cache-values.outputs.cache-hit != 'true'
run: echo "values" > values.txt
73 changes: 73 additions & 0 deletions acceptance/testdata/workflow/run-cancel.txtar
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Use gh as a credential helper
exec gh auth setup-git

# Create a repository with a file so it has a default branch
exec gh repo create $ORG/$SCRIPT_NAME-$RANDOM_STRING --add-readme --private

# Defer repo cleanup
defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING

# Clone the repo
exec gh repo clone $ORG/$SCRIPT_NAME-$RANDOM_STRING

# commit the workflow file
cd $SCRIPT_NAME-$RANDOM_STRING
mkdir .github/workflows
mv ../workflow.yml .github/workflows/workflow.yml
exec git add .github/workflows/workflow.yml
exec git commit -m 'Create workflow file'
exec git push -u origin main

# Sleep because it takes a second for the workflow to register
sleep 1

# Check the workflow is indeed created
exec gh workflow list
stdout 'Test Workflow Name'

# Run the workflow
exec gh workflow run 'Test Workflow Name'

# It takes some time for a workflow run to register
sleep 10

# Get the run ID we want to cancel
exec gh run list --json databaseId --jq '.[0].databaseId'
stdout2env RUN_ID

# cancel the workflow run
exec gh run cancel $RUN_ID
stdout '✓ Request to cancel workflow [0-9]+ submitted.'

# Wait for workflow to complete
exec gh run watch $RUN_ID

# Check the workflow run is cancelled
exec gh run list --json conclusion --jq '.[0].conclusion'
stdout 'cancelled'

-- workflow.yml --
# This is a basic workflow to help you get started with Actions

name: Test Workflow Name

# Controls when the workflow will run
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v4

# Runs a single command using the runners shell
- name: Run a one-line script
run: sleep 30
74 changes: 74 additions & 0 deletions acceptance/testdata/workflow/run-delete.txtar
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Use gh as a credential helper
exec gh auth setup-git

# Create a repository with a file so it has a default branch
exec gh repo create $ORG/$SCRIPT_NAME-$RANDOM_STRING --add-readme --private

# Defer repo cleanup
defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING

# Clone the repo
exec gh repo clone $ORG/$SCRIPT_NAME-$RANDOM_STRING

# commit the workflow file
cd $SCRIPT_NAME-$RANDOM_STRING
mkdir .github/workflows
mv ../workflow.yml .github/workflows/workflow.yml
exec git add .github/workflows/workflow.yml
exec git commit -m 'Create workflow file'
exec git push -u origin main

# Sleep because it takes a second for the workflow to register
sleep 1

# Check the workflow is indeed created
exec gh workflow list
stdout 'Test Workflow Name'

# Run the workflow
exec gh workflow run 'Test Workflow Name'

# It takes some time for a workflow run to register
sleep 10

# Get the run ID we want to watch & delete
exec gh run list --json databaseId --jq '.[0].databaseId'
stdout2env RUN_ID

# Wait for workflow to complete
exec gh run watch $RUN_ID --exit-status

# Delete the workflow run
exec gh run delete $RUN_ID
stdout '✓ Request to delete workflow submitted.'

# It takes some time for a workflow run to be deleted
sleep 5

# Check the workflow run is cancelled, which is implied by an empty list
exec gh run list
stdout ''

-- workflow.yml --
# This is a basic workflow to help you get started with Actions

name: Test Workflow Name

# Controls when the workflow will run
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest

# Steps represent a sequence of tasks that will be executed as part of the job
steps:

# Runs a single command using the runners shell
- name: Run a one-line script
run: echo Hello, world!
70 changes: 70 additions & 0 deletions acceptance/testdata/workflow/run-download.txtar
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Use gh as a credential helper
exec gh auth setup-git

# Create a repository with a file so it has a default branch
exec gh repo create $ORG/$SCRIPT_NAME-$RANDOM_STRING --add-readme --private

# Defer repo cleanup
defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING

# Clone the repo
exec gh repo clone $ORG/$SCRIPT_NAME-$RANDOM_STRING

# commit the workflow file
cd $SCRIPT_NAME-$RANDOM_STRING
mkdir .github/workflows
mv ../workflow.yml .github/workflows/workflow.yml
exec git add .github/workflows/workflow.yml
exec git commit -m 'Create workflow file'
exec git push -u origin main

# Sleep because it takes a second for the workflow to register
sleep 1

# Check the workflow is indeed created
exec gh workflow list
stdout 'Test Workflow Name'

# Run the workflow
exec gh workflow run 'Test Workflow Name'

# It takes some time for a workflow run to register
sleep 10

# Get the run ID we want to watch
exec gh run list --json databaseId --jq '.[0].databaseId'
stdout2env RUN_ID

# Wait for workflow to complete
exec gh run watch $RUN_ID --exit-status

# Download the artifact
exec gh run download $RUN_ID

# Check if we downloaded the artifact
exists ./my-artifact/world.txt

-- workflow.yml --
# This is a basic workflow to help you get started with Actions

name: Test Workflow Name

# Controls when the workflow will run
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- run: echo hello > world.txt
- uses: actions/upload-artifact@v4
with:
name: my-artifact
path: world.txt
Loading