|
Download docs/integration-tests.md from SaylorTwift/gemini-cli: direct link, hf CLI and curl.
- Browser
- Download file 8.53 kB
-
https://huggingface.co/SaylorTwift/gemini-cli/resolve/main/docs/integration-tests.md
- Command line
-
hf download hf://SaylorTwift/gemini-cli/docs/integration-tests.md
-
curl -L -o integration-tests.md https://huggingface.co/SaylorTwift/gemini-cli/resolve/main/docs/integration-tests.md
8.53 kB
| # Integration tests | |
| This document provides information about the integration testing framework used | |
| in this project. | |
| ## Overview | |
| The integration tests are designed to validate the end-to-end functionality of | |
| Gemini CLI. They execute the built binary in a controlled environment and verify | |
| that it behaves as expected when interacting with the file system. | |
| These tests are located in the `integration-tests` directory and are run using a | |
| custom test runner. | |
| ## Building the tests | |
| Prior to running any integration tests, you need to create a release bundle that | |
| you want to actually test: | |
| ```bash | |
| npm run bundle | |
| ``` | |
| You must re-run this command after making any changes to the CLI source code, | |
| but not after making changes to tests. | |
| ## Running the tests | |
| The integration tests are not run as part of the default `npm run test` command. | |
| They must be run explicitly using the `npm run test:integration:all` script. | |
| The integration tests can also be run using the following shortcut: | |
| ```bash | |
| npm run test:e2e | |
| ``` | |
| ## Running a specific set of tests | |
| To run a subset of test files, you can use | |
| `npm run <integration test command> <file_name1> ....` where <integration | |
| test command> is either `test:e2e` or `test:integration*` and `<file_name>` | |
| is any of the `.test.js` files in the `integration-tests/` directory. For | |
| example, the following command runs `list_directory.test.js` and | |
| `write_file.test.js`: | |
| ```bash | |
| npm run test:e2e list_directory write_file | |
| ``` | |
| ### Running a single test by name | |
| To run a single test by its name, use the `--test-name-pattern` flag: | |
| ```bash | |
| npm run test:e2e -- --test-name-pattern "reads a file" | |
| ``` | |
| ### Regenerating model responses | |
| Some integration tests use faked out model responses, which may need to be | |
| regenerated from time to time as the implementations change. | |
| To regenerate these golden files, set the REGENERATE_MODEL_GOLDENS environment | |
| variable to "true" when running the tests, for example: | |
| **WARNING**: If running locally you should review these updated responses for | |
| any information about yourself or your system that gemini may have included in | |
| these responses. | |
| ```bash | |
| REGENERATE_MODEL_GOLDENS="true" npm run test:e2e | |
| ``` | |
| **WARNING**: Make sure you run **await rig.cleanup()** at the end of your test, | |
| else the golden files will not be updated. | |
| ### Deflaking a test | |
| Before adding a **new** integration test, you should test it at least 5 times | |
| with the deflake script or workflow to make sure that it is not flaky. | |
| ### Deflake script | |
| ```bash | |
| npm run deflake -- --runs=5 --command="npm run test:e2e -- -- --test-name-pattern '<your-new-test-name>'" | |
| ``` | |
| #### Deflake workflow | |
| ```bash | |
| gh workflow run deflake.yml --ref <your-branch> -f test_name_pattern="<your-test-name-pattern>" | |
| ``` | |
| ### Running all tests | |
| To run the entire suite of integration tests, use the following command: | |
| ```bash | |
| npm run test:integration:all | |
| ``` | |
| ### Sandbox matrix | |
| The `all` command will run tests for `no sandboxing`, `docker` and `podman`. | |
| Each individual type can be run using the following commands: | |
| ```bash | |
| npm run test:integration:sandbox:none | |
| ``` | |
| ```bash | |
| npm run test:integration:sandbox:docker | |
| ``` | |
| ```bash | |
| npm run test:integration:sandbox:podman | |
| ``` | |
| ## Memory regression tests | |
| Memory regression tests are designed to detect heap growth and leaks across key | |
| CLI scenarios. They are located in the `memory-tests` directory. | |
| These tests are distinct from standard integration tests because they measure | |
| memory usage and compare it against committed baselines. | |
| ### Running memory tests | |
| Memory tests are not run as part of the default `npm run test` or | |
| `npm run test:e2e` commands. They are run nightly in CI but can be run manually: | |
| ```bash | |
| npm run test:memory | |
| ``` | |
| ### Updating baselines | |
| If you intentionally change behavior that affects memory usage, you may need to | |
| update the baselines. Set the `UPDATE_MEMORY_BASELINES` environment variable to | |
| `true`: | |
| ```bash | |
| UPDATE_MEMORY_BASELINES=true npm run test:memory | |
| ``` | |
| This will run the tests, take median snapshots, and overwrite | |
| `memory-tests/baselines.json`. You should review the changes and commit the | |
| updated baseline file. | |
| ### How it works | |
| The harness (`MemoryTestHarness` in `packages/test-utils`): | |
| - Forces garbage collection multiple times to reduce noise. | |
| - Takes median snapshots to filter spikes. | |
| - Compares against baselines with a 10% tolerance. | |
| - Can analyze sustained leaks across 3 snapshots using `analyzeSnapshots()`. | |
| ## Performance regression tests | |
| Performance regression tests are designed to detect wall-clock time, CPU usage, | |
| and event loop delay regressions across key CLI scenarios. They are located in | |
| the `perf-tests` directory. | |
| These tests are distinct from standard integration tests because they measure | |
| performance metrics and compare it against committed baselines. | |
| ### Running performance tests | |
| Performance tests are not run as part of the default `npm run test` or | |
| `npm run test:e2e` commands. They are run nightly in CI but can be run manually: | |
| ```bash | |
| npm run test:perf | |
| ``` | |
| ### Updating baselines | |
| If you intentionally change behavior that affects performance, you may need to | |
| update the baselines. Set the `UPDATE_PERF_BASELINES` environment variable to | |
| `true`: | |
| ```bash | |
| UPDATE_PERF_BASELINES=true npm run test:perf | |
| ``` | |
| This will run the tests multiple times (with warmup), apply IQR outlier | |
| filtering, and overwrite `perf-tests/baselines.json`. You should review the | |
| changes and commit the updated baseline file. | |
| ### How it works | |
| The harness (`PerfTestHarness` in `packages/test-utils`): | |
| - Measures wall-clock time using `performance.now()`. | |
| - Measures CPU usage using `process.cpuUsage()`. | |
| - Monitors event loop delay using `perf_hooks.monitorEventLoopDelay()`. | |
| - Applies IQR (Interquartile Range) filtering to remove outlier samples. | |
| - Compares against baselines with a 15% tolerance. | |
| ## Diagnostics | |
| The integration test runner provides several options for diagnostics to help | |
| track down test failures. | |
| ### Keeping test output | |
| You can preserve the temporary files created during a test run for inspection. | |
| This is useful for debugging issues with file system operations. | |
| To keep the test output set the `KEEP_OUTPUT` environment variable to `true`. | |
| ```bash | |
| KEEP_OUTPUT=true npm run test:integration:sandbox:none | |
| ``` | |
| When output is kept, the test runner will print the path to the unique directory | |
| for the test run. | |
| ### Verbose output | |
| For more detailed debugging, set the `VERBOSE` environment variable to `true`. | |
| ```bash | |
| VERBOSE=true npm run test:integration:sandbox:none | |
| ``` | |
| When using `VERBOSE=true` and `KEEP_OUTPUT=true` in the same command, the output | |
| is streamed to the console and also saved to a log file within the test's | |
| temporary directory. | |
| The verbose output is formatted to clearly identify the source of the logs: | |
| ``` | |
| --- TEST: <log dir>:<test-name> --- | |
| ... output from the gemini command ... | |
| --- END TEST: <log dir>:<test-name> --- | |
| ``` | |
| ## Linting and formatting | |
| To ensure code quality and consistency, the integration test files are linted as | |
| part of the main build process. You can also manually run the linter and | |
| auto-fixer. | |
| ### Running the linter | |
| To check for linting errors, run the following command: | |
| ```bash | |
| npm run lint | |
| ``` | |
| You can include the `:fix` flag in the command to automatically fix any fixable | |
| linting errors: | |
| ```bash | |
| npm run lint:fix | |
| ``` | |
| ## Directory structure | |
| The integration tests create a unique directory for each test run inside the | |
| `.integration-tests` directory. Within this directory, a subdirectory is created | |
| for each test file, and within that, a subdirectory is created for each | |
| individual test case. | |
| This structure makes it easy to locate the artifacts for a specific test run, | |
| file, or case. | |
| ``` | |
| .integration-tests/ | |
| βββ <run-id>/ | |
| βββ <test-file-name>.test.js/ | |
| βββ <test-case-name>/ | |
| βββ output.log | |
| βββ ...other test artifacts... | |
| ``` | |
| ## Continuous integration | |
| To ensure the integration tests are always run, a GitHub Actions workflow is | |
| defined in `.github/workflows/chained_e2e.yml`. This workflow automatically runs | |
| the integrations tests for pull requests against the `main` branch, or when a | |
| pull request is added to a merge queue. | |
| The workflow runs the tests in different sandboxing environments to ensure | |
| Gemini CLI is tested across each: | |
| - `sandbox:none`: Runs the tests without any sandboxing. | |
| - `sandbox:docker`: Runs the tests in a Docker container. | |
| - `sandbox:podman`: Runs the tests in a Podman container. | |