diff options
| author | CoprDistGit <infra@openeuler.org> | 2023-05-15 04:39:26 +0000 |
|---|---|---|
| committer | CoprDistGit <infra@openeuler.org> | 2023-05-15 04:39:26 +0000 |
| commit | fd71f638f35eaaacfc90b74fef1b5ea351fb4c5c (patch) | |
| tree | eaa89056661621b5b578cc210f67c31cb78462d2 /python-all-repos.spec | |
| parent | e5979869983711a17e54d80a82bb3ffb354f689d (diff) | |
automatic import of python-all-repos
Diffstat (limited to 'python-all-repos.spec')
| -rw-r--r-- | python-all-repos.spec | 1440 |
1 files changed, 1440 insertions, 0 deletions
diff --git a/python-all-repos.spec b/python-all-repos.spec new file mode 100644 index 0000000..5c2f68a --- /dev/null +++ b/python-all-repos.spec @@ -0,0 +1,1440 @@ +%global _empty_manifest_terminate_build 0 +Name: python-all-repos +Version: 1.25.0 +Release: 1 +Summary: Clone all your repositories and apply sweeping changes. +License: MIT +URL: https://github.com/asottile/all-repos +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/00/54/c9f28df541c717bbe1e1920ddb99a4a52c6a2accd69ce56602632b2f03c3/all_repos-1.25.0.tar.gz +BuildArch: noarch + +Requires: python3-identify +Requires: python3-packaging +Requires: python3-contextlib-chdir + +%description +Clone all your repositories and apply sweeping changes. +## Installation +```bash +pip install all-repos +``` +## CLI +All command line interfaces provided by `all-repos` provide the following +options: +- `-h` / `--help`: show usage information +- `-C CONFIG_FILENAME` / `--config-filename CONFIG_FILENAME`: use a non-default + config file (the default `all-repos.json` can be changed with the environment + variable `ALL_REPOS_CONFIG_FILENAME`). +- `--color {auto,always,never}`: use color in output (default `auto`). +### `all-repos-complete [options]` +Add `git clone` tab completion for all-repos repositories. +Requires [jq](https://stedolan.github.io/jq/) to function. +Add to `.bash_profile`: +```bash +eval "$(all-repos-complete -C ~/.../all-repos.json --bash)" +``` +### `all-repos-clone [options]` +Clone all the repositories into the `output_dir`. If run again, this command +will update existing repositories. +Options: +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `8`). +Sample invocations: +- `all-repos-clone`: clone the repositories specified in `all-repos.json` +- `all-repos-clone -C all-repos2.json`: clone using a non-default config + filename. +### `all-repos-find-files [options] PATTERN` +Similar to a distributed `git ls-files | grep -P PATTERN`. +Arguments: +- `PATTERN`: the [python regex](https://docs.python.org/3/library/re.html) + to match. +Options: +- `--repos-with-matches`: only print repositories with matches. +Sample invocations: +- `all-repos-find-files setup.py`: find all `setup.py` files. +- `all-repos-find-files --repos setup.py`: find all repositories containing + a `setup.py`. +### `all-repos-grep [options] [GIT_GREP_OPTIONS]` +Similar to a distributed `git grep ...`. +Options: +- `--repos-with-matches`: only print repositories with matches. +- `GIT_GREP_OPTIONS`: additional arguments will be passed on to `git grep`. + see `git grep --help` for available options. +Sample invocations: +- `all-repos-grep pre-commit -- 'requirements*.txt'`: find all repositories + which have `pre-commit` listed in a requirements file. +- `all-repos-grep -L six -- setup.py`: find setup.py files which do not + contain `six`. +### `all-repos-list-repos [options]` +List all cloned repository names. +### `all-repos-manual [options]` +Interactively apply a manual change across repos. +_note_: `all-repos-manual` will always run in `--interactive` autofixing mode. +_note_: `all-repos-manual` _requires_ the `--repos` autofixer option. +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-manual` is + an autofixer and supports all of the autofixer options. +- `--branch-name BRANCH_NAME`: override the autofixer branch name (default + `all-repos-manual`). +- `--commit-msg COMMIT_MSG` (required): set the autofixer commit message. +### `all-repos-sed [options] EXPRESSION FILENAMES` +Similar to a distributed +`git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION`. +_note_: this assumes GNU sed. If you're on macOS, install `gnu-sed` with Homebrew: +```bash +brew install gnu-sed +# Add to .bashrc / .zshrc +export PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH" +``` +Arguments: +- `EXPRESSION`: sed program. For example: `s/hi/hello/g`. +- `FILENAMES`: filenames glob (passed to `git ls-files`). +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-sed` is + an autofixer and supports all of the autofixer options. +- `-r` / `--regexp-extended`: use extended regular expressions in the script. + See `man sed` for further details. +- `--branch-name BRANCH_NAME` override the autofixer branch name (default + `all-repos-sed`). +- `--commit-msg COMMIT_MSG` override the autofixer commit message. (default + `git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION`). +Sample invocations: +- `all-repos-sed 's/foo/bar/g' -- '*'`: replace `foo` with `bar` in all files. +## Configuring +A configuration file looks roughly like this: +```json +{ + "output_dir": "output", + "source": "all_repos.source.github", + "source_settings": { + "api_key": "...", + "username": "asottile" + }, + "push": "all_repos.push.github_pull_request", + "push_settings": { + "api_key": "...", + "username": "asottile" + } +} +``` +- `output_dir`: where repositories will be cloned to when `all-repos-clone` is + run. +- `source`: the module import path to a `source`, see below for builtin + source modules as well as directions for writing your own. +- `source_settings`: the source-type-specific settings, the source module's + documentation will explain the various possible values. +- `push`: the module import path to a `push`, see below for builtin push + modules as well as directions for writing your own. +- `push_settings`: the push-type-specific settings, the push module's + documentation will explain the various possible values. +- `include` (default `""`): python regex for selecting repositories. Only + repository names which match this regex will be included. +- `exclude` (default `"^$"`): python regex for excluding repositories. + Repository names which match this regex will be excluded. +- `all_branches` (default `false`): whether to clone all of the branches or + just the default upstream branch. +## Source modules +### `all_repos.source.json_file` +Clones all repositories listed in a file. The file must be formatted as +follows: +```json +{ + "example/repo1": "https://git.example.com/example/repo1", + "repo2": "https://git.example.com/repo2" +} +``` +#### Required `source_settings` +- `filename`: file containing repositories one-per-line. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_key1}/ ++--- {repo_key2}/ ++--- {repo_key3}/ +``` +### `all_repos.source.github` +Clones all repositories available to a user on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `source_settings` +- `collaborator` (default `false`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.github_forks` +Clones all repositories forked from a repository on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `repo`: the repo which has forks +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `true`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.github_org` +Clones all repositories from an organization on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.gitolite` +Clones all repositories available to a user on a +[gitolite](http://gitolite.com/gitolite/index.html) host. +#### Required `source_settings` +- `username`: the user to SSH to the server as (usually `git`) +- `hostname`: the hostname of your gitolite server (e.g. `git.mycompany.com`) +The gitolite API is served over SSH. It is assumed that when `all-repos-clone` +is called, it's possible to make SSH connections with the username and hostname +configured here in order to query that API. +#### Optional `source_settings` +- `mirror_path` (default `None`): an optional mirror to clone repositories from. + This is a Python format string, and can use the variable `repo_name`. + This can be anything git understands, such as another remote server (e.g. + `gitmirror.mycompany.com:{repo_name}`) or a local path (e.g. + `/gitolite/git/{repo_name}.git`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_name1}.git/ ++--- {repo_name2}.git/ ++--- {repo_name3}.git/ +``` +### `all_repos.source.bitbucket` +Clones all repositories available to a user on Bitbucket Cloud. +#### Required `source_settings` +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.org/account/admin/app-passwords). + - We need the scope: Repositories -> Read +### `all_repos.source.bitbucket_server` +Clones all repositories available to a user on Bitbucket Server. +#### Required `source_settings` +- `base_url`: the bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +#### Optional `source_settings` +- `project` (default `None`): an optional project to restrict the search for repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.gitlab_org` +Clones all repositories from an organization on gitlab. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `read_api`, `read_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `base_url`: (default `https://gitlab.com/api/v4`) the gitlab server URL +- `archived` (default: `false`): whether to include archived repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {org}/ + +--- {subpgroup1}/ + +--- {subpgroup2}/ + +--- {repo1}/ + +--- {repo2}/ + +--- {repo3}/ + +--- {repo4}/ +``` +## Writing your own source +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the +`source_settings` dictionary. +An easy way to implement the `Settings` class is by using a `namedtuple`: +```python +Settings = collections.namedtuple('Settings', ('required_thing', 'optional')) +Settings.__new__.__defaults__ = ('optional default value',) +``` +In this example, the `required_thing` setting is a **required** setting +whereas `optional` may be omitted (and will get a default value of +`'optional default value'`). +### `def list_repos(settings: Settings) -> Dict[str, str]:` callable +This callable will be passed an instance of your `Settings` class. It must +return a mapping from `{repo_name: repository_url}`. The `repo_name` is the +directory name inside the `output_dir`. +## Push modules +### `all_repos.push.merge_to_master` +Merges the branch directly to the default branch and pushes. The commands it +runs look roughly like this: +```bash +git checkout main +git pull +git merge --no-ff $BRANCH +git push origin HEAD +``` +#### Optional `push_settings` +- `fast_forward` (default: `false`): if `true`, perform a fast-forward + merge (`--ff-only`). If `false`, create a merge commit (`--no-ff`). +### `all_repos.push.github_pull_request` +Pushes the branch to `origin` and then creates a github pull request for the +branch. +#### Required `push_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `push_settings` +- `fork` (default: `false`): (if applicable) a fork will be created and pushed + to instead of the upstream repository. The pull request will then be made + to the upstream repository. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +### `all_repos.push.bitbucket_server_pull_request` +Pushes the branch to `origin` and then creates a Bitbucket pull request for the branch. +#### Required `push_settings` +- `base_url`: the Bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +### `all_repos.push.gitlab_pull_request` +Pushes the branch to `origin` and then creates a GitLab pull request for the branch. +#### Required `push_settings` +- `base_url`: the GitLab server URL (eg `https://{gitlab.domain.com}/api/v4`) +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `write_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +### `all_repos.push.readonly` +Does nothing. +#### `push_settings` +There are no configurable settings for `readonly`. +## Writing your own push module +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the `push_settings` +dictionary. +### `def push(settings: Settings, branch_name: str) -> None:` +This callable will be passed an instance of your `Settings` class. It should +deploy the branch. The function will be called with the root of the +repository as the `cwd`. +## Writing an autofixer +An autofixer applies a change over all repositories. +`all-repos` provides several api functions to write your autofixers with: +### `all_repos.autofix_lib.add_fixer_args` +```python +def add_fixer_args(parser): +``` +Adds the autofixer cli options. +Options: +- `--dry-run`: show what would happen but do not push. +- `-i` / `--interactive`: interactively approve / deny fixes. +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `1`). +- `--limit LIMIT`: maximum number of repos to process (default: unlimited). +- `--author AUTHOR`: override commit author. This is passed directly to + `git commit`. An example: `--author='Herp Derp <herp.derp@umich.edu>'`. +- `--repos [REPOS [REPOS ...]]`: run against specific repositories instead. + This is especially useful with `xargs autofixer ... --repos`. This can be + used to specify repositories which are not managed by `all-repos`. +### `all_repos.autofix_lib.from_cli` +```python +def from_cli(args, *, find_repos, msg, branch_name): +``` +Parse cli arguments and produce `autofix_lib` primitives. Returns +`(repos, config, commit, autofix_settings)`. This is handled separately from +`fix` to allow for fixers to adjust arguments. +- `find_repos`: callback taking `Config` as a positional argument. +- `msg`: commit message. +- `branch_name`: identifier used to construct the branch name. +### `all_repos.autofix_lib.fix` +```python +def fix( + repos, *, + apply_fix, + check_fix=_noop_check_fix, + config: Config, + commit: Commit, + autofix_settings: AutofixSettings, +): +``` +Apply the fix. +- `apply_fix`: callback which will be called once per repository. The `cwd` + when the function is called will be the root of the repository. +### `all_repos.autofix_lib.run` +```python +def run(*cmd, **kwargs): +``` +Wrapper around `subprocess.run` which prints the command it will run. Unlike +`subprocess.run`, this defaults `check=True` unless explicitly disabled. +### Example autofixer +The trivial autofixer is as follows: +```python +import argparse +from all_repos import autofix_lib +def find_repos(config): + return [] +def apply_fix(): + pass +def main(argv=None): + parser = argparse.ArgumentParser() + autofix_lib.add_fixer_args(parser) + args = parser.parse_args(argv) + repos, config, commit, autofix_settings = autofix_lib.from_cli( + args, find_repos=find_repos, msg='msg', branch_name='branch-name', + ) + autofix_lib.fix( + repos, apply_fix=apply_fix, config=config, commit=commit, + autofix_settings=autofix_settings, + ) +if __name__ == '__main__': + raise SystemExit(main()) +``` +You can find some more involved examples in [all_repos/autofix](https://github.com/asottile/all-repos/tree/main/all_repos/autofix): +- `all_repos.autofix.azure_pipelines_autoupdate`: upgrade pinned azure + pipelines template repository references. +- `all_repos.autofix.pre_commit_autoupdate`: runs `pre-commit autoupdate`. +- `all_repos.autofix.pre_commit_autopep8_migrate`: migrates `autopep8-wrapper` + from [pre-commit/pre-commit-hooks] to [mirrors-autopep8]. +- `all_repos.autofix.pre_commit_cache_dir`: updates the cache directory + for travis-ci / appveyor for pre-commit 1.x. +- `all_repos.autofix.pre_commit_flake8_migrate`: migrates `flake8` from + [pre-commit/pre-commit-hooks] to [pycqa/flake8]. +- `all_repos.autofix.pre_commit_migrate_config`: runs + `pre-commit migrate-config`. +- `all_repos.autofix.setup_py_upgrade`: runs [setup-py-upgrade] and then + [setup-cfg-fmt] to migrate `setup.py` to `setup.cfg`. +[pre-commit/pre-commit-hooks]: https://github.com/pre-commit/pre-commit-hooks +[mirrors-autopep8]: https://github.com/pre-commit/mirrors-autopep8 +[pycqa/flake8]: https://gitlab.com/pycqa/flake8 +[setup-py-upgrade]: https://github.com/asottile/setup-py-upgrade +[setup-cfg-fmt]: https://github.com/asottile/setup-cfg-fmt + +%package -n python3-all-repos +Summary: Clone all your repositories and apply sweeping changes. +Provides: python-all-repos +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-all-repos +Clone all your repositories and apply sweeping changes. +## Installation +```bash +pip install all-repos +``` +## CLI +All command line interfaces provided by `all-repos` provide the following +options: +- `-h` / `--help`: show usage information +- `-C CONFIG_FILENAME` / `--config-filename CONFIG_FILENAME`: use a non-default + config file (the default `all-repos.json` can be changed with the environment + variable `ALL_REPOS_CONFIG_FILENAME`). +- `--color {auto,always,never}`: use color in output (default `auto`). +### `all-repos-complete [options]` +Add `git clone` tab completion for all-repos repositories. +Requires [jq](https://stedolan.github.io/jq/) to function. +Add to `.bash_profile`: +```bash +eval "$(all-repos-complete -C ~/.../all-repos.json --bash)" +``` +### `all-repos-clone [options]` +Clone all the repositories into the `output_dir`. If run again, this command +will update existing repositories. +Options: +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `8`). +Sample invocations: +- `all-repos-clone`: clone the repositories specified in `all-repos.json` +- `all-repos-clone -C all-repos2.json`: clone using a non-default config + filename. +### `all-repos-find-files [options] PATTERN` +Similar to a distributed `git ls-files | grep -P PATTERN`. +Arguments: +- `PATTERN`: the [python regex](https://docs.python.org/3/library/re.html) + to match. +Options: +- `--repos-with-matches`: only print repositories with matches. +Sample invocations: +- `all-repos-find-files setup.py`: find all `setup.py` files. +- `all-repos-find-files --repos setup.py`: find all repositories containing + a `setup.py`. +### `all-repos-grep [options] [GIT_GREP_OPTIONS]` +Similar to a distributed `git grep ...`. +Options: +- `--repos-with-matches`: only print repositories with matches. +- `GIT_GREP_OPTIONS`: additional arguments will be passed on to `git grep`. + see `git grep --help` for available options. +Sample invocations: +- `all-repos-grep pre-commit -- 'requirements*.txt'`: find all repositories + which have `pre-commit` listed in a requirements file. +- `all-repos-grep -L six -- setup.py`: find setup.py files which do not + contain `six`. +### `all-repos-list-repos [options]` +List all cloned repository names. +### `all-repos-manual [options]` +Interactively apply a manual change across repos. +_note_: `all-repos-manual` will always run in `--interactive` autofixing mode. +_note_: `all-repos-manual` _requires_ the `--repos` autofixer option. +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-manual` is + an autofixer and supports all of the autofixer options. +- `--branch-name BRANCH_NAME`: override the autofixer branch name (default + `all-repos-manual`). +- `--commit-msg COMMIT_MSG` (required): set the autofixer commit message. +### `all-repos-sed [options] EXPRESSION FILENAMES` +Similar to a distributed +`git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION`. +_note_: this assumes GNU sed. If you're on macOS, install `gnu-sed` with Homebrew: +```bash +brew install gnu-sed +# Add to .bashrc / .zshrc +export PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH" +``` +Arguments: +- `EXPRESSION`: sed program. For example: `s/hi/hello/g`. +- `FILENAMES`: filenames glob (passed to `git ls-files`). +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-sed` is + an autofixer and supports all of the autofixer options. +- `-r` / `--regexp-extended`: use extended regular expressions in the script. + See `man sed` for further details. +- `--branch-name BRANCH_NAME` override the autofixer branch name (default + `all-repos-sed`). +- `--commit-msg COMMIT_MSG` override the autofixer commit message. (default + `git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION`). +Sample invocations: +- `all-repos-sed 's/foo/bar/g' -- '*'`: replace `foo` with `bar` in all files. +## Configuring +A configuration file looks roughly like this: +```json +{ + "output_dir": "output", + "source": "all_repos.source.github", + "source_settings": { + "api_key": "...", + "username": "asottile" + }, + "push": "all_repos.push.github_pull_request", + "push_settings": { + "api_key": "...", + "username": "asottile" + } +} +``` +- `output_dir`: where repositories will be cloned to when `all-repos-clone` is + run. +- `source`: the module import path to a `source`, see below for builtin + source modules as well as directions for writing your own. +- `source_settings`: the source-type-specific settings, the source module's + documentation will explain the various possible values. +- `push`: the module import path to a `push`, see below for builtin push + modules as well as directions for writing your own. +- `push_settings`: the push-type-specific settings, the push module's + documentation will explain the various possible values. +- `include` (default `""`): python regex for selecting repositories. Only + repository names which match this regex will be included. +- `exclude` (default `"^$"`): python regex for excluding repositories. + Repository names which match this regex will be excluded. +- `all_branches` (default `false`): whether to clone all of the branches or + just the default upstream branch. +## Source modules +### `all_repos.source.json_file` +Clones all repositories listed in a file. The file must be formatted as +follows: +```json +{ + "example/repo1": "https://git.example.com/example/repo1", + "repo2": "https://git.example.com/repo2" +} +``` +#### Required `source_settings` +- `filename`: file containing repositories one-per-line. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_key1}/ ++--- {repo_key2}/ ++--- {repo_key3}/ +``` +### `all_repos.source.github` +Clones all repositories available to a user on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `source_settings` +- `collaborator` (default `false`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.github_forks` +Clones all repositories forked from a repository on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `repo`: the repo which has forks +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `true`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.github_org` +Clones all repositories from an organization on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.gitolite` +Clones all repositories available to a user on a +[gitolite](http://gitolite.com/gitolite/index.html) host. +#### Required `source_settings` +- `username`: the user to SSH to the server as (usually `git`) +- `hostname`: the hostname of your gitolite server (e.g. `git.mycompany.com`) +The gitolite API is served over SSH. It is assumed that when `all-repos-clone` +is called, it's possible to make SSH connections with the username and hostname +configured here in order to query that API. +#### Optional `source_settings` +- `mirror_path` (default `None`): an optional mirror to clone repositories from. + This is a Python format string, and can use the variable `repo_name`. + This can be anything git understands, such as another remote server (e.g. + `gitmirror.mycompany.com:{repo_name}`) or a local path (e.g. + `/gitolite/git/{repo_name}.git`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_name1}.git/ ++--- {repo_name2}.git/ ++--- {repo_name3}.git/ +``` +### `all_repos.source.bitbucket` +Clones all repositories available to a user on Bitbucket Cloud. +#### Required `source_settings` +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.org/account/admin/app-passwords). + - We need the scope: Repositories -> Read +### `all_repos.source.bitbucket_server` +Clones all repositories available to a user on Bitbucket Server. +#### Required `source_settings` +- `base_url`: the bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +#### Optional `source_settings` +- `project` (default `None`): an optional project to restrict the search for repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.gitlab_org` +Clones all repositories from an organization on gitlab. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `read_api`, `read_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `base_url`: (default `https://gitlab.com/api/v4`) the gitlab server URL +- `archived` (default: `false`): whether to include archived repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {org}/ + +--- {subpgroup1}/ + +--- {subpgroup2}/ + +--- {repo1}/ + +--- {repo2}/ + +--- {repo3}/ + +--- {repo4}/ +``` +## Writing your own source +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the +`source_settings` dictionary. +An easy way to implement the `Settings` class is by using a `namedtuple`: +```python +Settings = collections.namedtuple('Settings', ('required_thing', 'optional')) +Settings.__new__.__defaults__ = ('optional default value',) +``` +In this example, the `required_thing` setting is a **required** setting +whereas `optional` may be omitted (and will get a default value of +`'optional default value'`). +### `def list_repos(settings: Settings) -> Dict[str, str]:` callable +This callable will be passed an instance of your `Settings` class. It must +return a mapping from `{repo_name: repository_url}`. The `repo_name` is the +directory name inside the `output_dir`. +## Push modules +### `all_repos.push.merge_to_master` +Merges the branch directly to the default branch and pushes. The commands it +runs look roughly like this: +```bash +git checkout main +git pull +git merge --no-ff $BRANCH +git push origin HEAD +``` +#### Optional `push_settings` +- `fast_forward` (default: `false`): if `true`, perform a fast-forward + merge (`--ff-only`). If `false`, create a merge commit (`--no-ff`). +### `all_repos.push.github_pull_request` +Pushes the branch to `origin` and then creates a github pull request for the +branch. +#### Required `push_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `push_settings` +- `fork` (default: `false`): (if applicable) a fork will be created and pushed + to instead of the upstream repository. The pull request will then be made + to the upstream repository. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +### `all_repos.push.bitbucket_server_pull_request` +Pushes the branch to `origin` and then creates a Bitbucket pull request for the branch. +#### Required `push_settings` +- `base_url`: the Bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +### `all_repos.push.gitlab_pull_request` +Pushes the branch to `origin` and then creates a GitLab pull request for the branch. +#### Required `push_settings` +- `base_url`: the GitLab server URL (eg `https://{gitlab.domain.com}/api/v4`) +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `write_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +### `all_repos.push.readonly` +Does nothing. +#### `push_settings` +There are no configurable settings for `readonly`. +## Writing your own push module +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the `push_settings` +dictionary. +### `def push(settings: Settings, branch_name: str) -> None:` +This callable will be passed an instance of your `Settings` class. It should +deploy the branch. The function will be called with the root of the +repository as the `cwd`. +## Writing an autofixer +An autofixer applies a change over all repositories. +`all-repos` provides several api functions to write your autofixers with: +### `all_repos.autofix_lib.add_fixer_args` +```python +def add_fixer_args(parser): +``` +Adds the autofixer cli options. +Options: +- `--dry-run`: show what would happen but do not push. +- `-i` / `--interactive`: interactively approve / deny fixes. +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `1`). +- `--limit LIMIT`: maximum number of repos to process (default: unlimited). +- `--author AUTHOR`: override commit author. This is passed directly to + `git commit`. An example: `--author='Herp Derp <herp.derp@umich.edu>'`. +- `--repos [REPOS [REPOS ...]]`: run against specific repositories instead. + This is especially useful with `xargs autofixer ... --repos`. This can be + used to specify repositories which are not managed by `all-repos`. +### `all_repos.autofix_lib.from_cli` +```python +def from_cli(args, *, find_repos, msg, branch_name): +``` +Parse cli arguments and produce `autofix_lib` primitives. Returns +`(repos, config, commit, autofix_settings)`. This is handled separately from +`fix` to allow for fixers to adjust arguments. +- `find_repos`: callback taking `Config` as a positional argument. +- `msg`: commit message. +- `branch_name`: identifier used to construct the branch name. +### `all_repos.autofix_lib.fix` +```python +def fix( + repos, *, + apply_fix, + check_fix=_noop_check_fix, + config: Config, + commit: Commit, + autofix_settings: AutofixSettings, +): +``` +Apply the fix. +- `apply_fix`: callback which will be called once per repository. The `cwd` + when the function is called will be the root of the repository. +### `all_repos.autofix_lib.run` +```python +def run(*cmd, **kwargs): +``` +Wrapper around `subprocess.run` which prints the command it will run. Unlike +`subprocess.run`, this defaults `check=True` unless explicitly disabled. +### Example autofixer +The trivial autofixer is as follows: +```python +import argparse +from all_repos import autofix_lib +def find_repos(config): + return [] +def apply_fix(): + pass +def main(argv=None): + parser = argparse.ArgumentParser() + autofix_lib.add_fixer_args(parser) + args = parser.parse_args(argv) + repos, config, commit, autofix_settings = autofix_lib.from_cli( + args, find_repos=find_repos, msg='msg', branch_name='branch-name', + ) + autofix_lib.fix( + repos, apply_fix=apply_fix, config=config, commit=commit, + autofix_settings=autofix_settings, + ) +if __name__ == '__main__': + raise SystemExit(main()) +``` +You can find some more involved examples in [all_repos/autofix](https://github.com/asottile/all-repos/tree/main/all_repos/autofix): +- `all_repos.autofix.azure_pipelines_autoupdate`: upgrade pinned azure + pipelines template repository references. +- `all_repos.autofix.pre_commit_autoupdate`: runs `pre-commit autoupdate`. +- `all_repos.autofix.pre_commit_autopep8_migrate`: migrates `autopep8-wrapper` + from [pre-commit/pre-commit-hooks] to [mirrors-autopep8]. +- `all_repos.autofix.pre_commit_cache_dir`: updates the cache directory + for travis-ci / appveyor for pre-commit 1.x. +- `all_repos.autofix.pre_commit_flake8_migrate`: migrates `flake8` from + [pre-commit/pre-commit-hooks] to [pycqa/flake8]. +- `all_repos.autofix.pre_commit_migrate_config`: runs + `pre-commit migrate-config`. +- `all_repos.autofix.setup_py_upgrade`: runs [setup-py-upgrade] and then + [setup-cfg-fmt] to migrate `setup.py` to `setup.cfg`. +[pre-commit/pre-commit-hooks]: https://github.com/pre-commit/pre-commit-hooks +[mirrors-autopep8]: https://github.com/pre-commit/mirrors-autopep8 +[pycqa/flake8]: https://gitlab.com/pycqa/flake8 +[setup-py-upgrade]: https://github.com/asottile/setup-py-upgrade +[setup-cfg-fmt]: https://github.com/asottile/setup-cfg-fmt + +%package help +Summary: Development documents and examples for all-repos +Provides: python3-all-repos-doc +%description help +Clone all your repositories and apply sweeping changes. +## Installation +```bash +pip install all-repos +``` +## CLI +All command line interfaces provided by `all-repos` provide the following +options: +- `-h` / `--help`: show usage information +- `-C CONFIG_FILENAME` / `--config-filename CONFIG_FILENAME`: use a non-default + config file (the default `all-repos.json` can be changed with the environment + variable `ALL_REPOS_CONFIG_FILENAME`). +- `--color {auto,always,never}`: use color in output (default `auto`). +### `all-repos-complete [options]` +Add `git clone` tab completion for all-repos repositories. +Requires [jq](https://stedolan.github.io/jq/) to function. +Add to `.bash_profile`: +```bash +eval "$(all-repos-complete -C ~/.../all-repos.json --bash)" +``` +### `all-repos-clone [options]` +Clone all the repositories into the `output_dir`. If run again, this command +will update existing repositories. +Options: +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `8`). +Sample invocations: +- `all-repos-clone`: clone the repositories specified in `all-repos.json` +- `all-repos-clone -C all-repos2.json`: clone using a non-default config + filename. +### `all-repos-find-files [options] PATTERN` +Similar to a distributed `git ls-files | grep -P PATTERN`. +Arguments: +- `PATTERN`: the [python regex](https://docs.python.org/3/library/re.html) + to match. +Options: +- `--repos-with-matches`: only print repositories with matches. +Sample invocations: +- `all-repos-find-files setup.py`: find all `setup.py` files. +- `all-repos-find-files --repos setup.py`: find all repositories containing + a `setup.py`. +### `all-repos-grep [options] [GIT_GREP_OPTIONS]` +Similar to a distributed `git grep ...`. +Options: +- `--repos-with-matches`: only print repositories with matches. +- `GIT_GREP_OPTIONS`: additional arguments will be passed on to `git grep`. + see `git grep --help` for available options. +Sample invocations: +- `all-repos-grep pre-commit -- 'requirements*.txt'`: find all repositories + which have `pre-commit` listed in a requirements file. +- `all-repos-grep -L six -- setup.py`: find setup.py files which do not + contain `six`. +### `all-repos-list-repos [options]` +List all cloned repository names. +### `all-repos-manual [options]` +Interactively apply a manual change across repos. +_note_: `all-repos-manual` will always run in `--interactive` autofixing mode. +_note_: `all-repos-manual` _requires_ the `--repos` autofixer option. +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-manual` is + an autofixer and supports all of the autofixer options. +- `--branch-name BRANCH_NAME`: override the autofixer branch name (default + `all-repos-manual`). +- `--commit-msg COMMIT_MSG` (required): set the autofixer commit message. +### `all-repos-sed [options] EXPRESSION FILENAMES` +Similar to a distributed +`git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION`. +_note_: this assumes GNU sed. If you're on macOS, install `gnu-sed` with Homebrew: +```bash +brew install gnu-sed +# Add to .bashrc / .zshrc +export PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH" +``` +Arguments: +- `EXPRESSION`: sed program. For example: `s/hi/hello/g`. +- `FILENAMES`: filenames glob (passed to `git ls-files`). +Options: +- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-sed` is + an autofixer and supports all of the autofixer options. +- `-r` / `--regexp-extended`: use extended regular expressions in the script. + See `man sed` for further details. +- `--branch-name BRANCH_NAME` override the autofixer branch name (default + `all-repos-sed`). +- `--commit-msg COMMIT_MSG` override the autofixer commit message. (default + `git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION`). +Sample invocations: +- `all-repos-sed 's/foo/bar/g' -- '*'`: replace `foo` with `bar` in all files. +## Configuring +A configuration file looks roughly like this: +```json +{ + "output_dir": "output", + "source": "all_repos.source.github", + "source_settings": { + "api_key": "...", + "username": "asottile" + }, + "push": "all_repos.push.github_pull_request", + "push_settings": { + "api_key": "...", + "username": "asottile" + } +} +``` +- `output_dir`: where repositories will be cloned to when `all-repos-clone` is + run. +- `source`: the module import path to a `source`, see below for builtin + source modules as well as directions for writing your own. +- `source_settings`: the source-type-specific settings, the source module's + documentation will explain the various possible values. +- `push`: the module import path to a `push`, see below for builtin push + modules as well as directions for writing your own. +- `push_settings`: the push-type-specific settings, the push module's + documentation will explain the various possible values. +- `include` (default `""`): python regex for selecting repositories. Only + repository names which match this regex will be included. +- `exclude` (default `"^$"`): python regex for excluding repositories. + Repository names which match this regex will be excluded. +- `all_branches` (default `false`): whether to clone all of the branches or + just the default upstream branch. +## Source modules +### `all_repos.source.json_file` +Clones all repositories listed in a file. The file must be formatted as +follows: +```json +{ + "example/repo1": "https://git.example.com/example/repo1", + "repo2": "https://git.example.com/repo2" +} +``` +#### Required `source_settings` +- `filename`: file containing repositories one-per-line. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_key1}/ ++--- {repo_key2}/ ++--- {repo_key3}/ +``` +### `all_repos.source.github` +Clones all repositories available to a user on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `source_settings` +- `collaborator` (default `false`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.github_forks` +Clones all repositories forked from a repository on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `repo`: the repo which has forks +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `true`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.github_org` +Clones all repositories from an organization on github. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `collaborator` (default `true`): whether to include repositories which are + not owned but can be contributed to as a collaborator. +- `forks` (default `false`): whether to include repositories which are forks. +- `private` (default `false`): whether to include private repositories. +- `archived` (default: `false`): whether to include archived repositories. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +#### Directory location +See the directory structure for +[`all_repos.source.github`](#all_repossourcegithub). +### `all_repos.source.gitolite` +Clones all repositories available to a user on a +[gitolite](http://gitolite.com/gitolite/index.html) host. +#### Required `source_settings` +- `username`: the user to SSH to the server as (usually `git`) +- `hostname`: the hostname of your gitolite server (e.g. `git.mycompany.com`) +The gitolite API is served over SSH. It is assumed that when `all-repos-clone` +is called, it's possible to make SSH connections with the username and hostname +configured here in order to query that API. +#### Optional `source_settings` +- `mirror_path` (default `None`): an optional mirror to clone repositories from. + This is a Python format string, and can use the variable `repo_name`. + This can be anything git understands, such as another remote server (e.g. + `gitmirror.mycompany.com:{repo_name}`) or a local path (e.g. + `/gitolite/git/{repo_name}.git`). +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {repo_name1}.git/ ++--- {repo_name2}.git/ ++--- {repo_name3}.git/ +``` +### `all_repos.source.bitbucket` +Clones all repositories available to a user on Bitbucket Cloud. +#### Required `source_settings` +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.org/account/admin/app-passwords). + - We need the scope: Repositories -> Read +### `all_repos.source.bitbucket_server` +Clones all repositories available to a user on Bitbucket Server. +#### Required `source_settings` +- `base_url`: the bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +#### Optional `source_settings` +- `project` (default `None`): an optional project to restrict the search for repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {username1}/ + +--- {repo1}/ + +--- {repo2}/ ++--- {username2}/ + +--- {repo3}/ +``` +### `all_repos.source.gitlab_org` +Clones all repositories from an organization on gitlab. +#### Required `source_settings` +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `read_api`, `read_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `org`: the organization to clone from +#### Optional `source_settings` +- `base_url`: (default `https://gitlab.com/api/v4`) the gitlab server URL +- `archived` (default: `false`): whether to include archived repositories. +#### Directory location +``` +output/ ++--- repos.json ++--- repos_filtered.json ++--- {org}/ + +--- {subpgroup1}/ + +--- {subpgroup2}/ + +--- {repo1}/ + +--- {repo2}/ + +--- {repo3}/ + +--- {repo4}/ +``` +## Writing your own source +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the +`source_settings` dictionary. +An easy way to implement the `Settings` class is by using a `namedtuple`: +```python +Settings = collections.namedtuple('Settings', ('required_thing', 'optional')) +Settings.__new__.__defaults__ = ('optional default value',) +``` +In this example, the `required_thing` setting is a **required** setting +whereas `optional` may be omitted (and will get a default value of +`'optional default value'`). +### `def list_repos(settings: Settings) -> Dict[str, str]:` callable +This callable will be passed an instance of your `Settings` class. It must +return a mapping from `{repo_name: repository_url}`. The `repo_name` is the +directory name inside the `output_dir`. +## Push modules +### `all_repos.push.merge_to_master` +Merges the branch directly to the default branch and pushes. The commands it +runs look roughly like this: +```bash +git checkout main +git pull +git merge --no-ff $BRANCH +git push origin HEAD +``` +#### Optional `push_settings` +- `fast_forward` (default: `false`): if `true`, perform a fast-forward + merge (`--ff-only`). If `false`, create a merge commit (`--no-ff`). +### `all_repos.push.github_pull_request` +Pushes the branch to `origin` and then creates a github pull request for the +branch. +#### Required `push_settings` +- `api_key`: the api key which the user will log in as. + - Use [the settings tab](//github.com/settings/tokens/new) to create a + personal access token. + - The minimum scope required to function is `public_repo`, though you'll + need `repo` to access private repositories. +- `api_key_env`: alternatively API key can also be passed via an environment variable +- `username`: the github username you will log in as. +#### Optional `push_settings` +- `fork` (default: `false`): (if applicable) a fork will be created and pushed + to instead of the upstream repository. The pull request will then be made + to the upstream repository. +- `base_url` (default: `https://api.github.com`) is the base URL to the Github + API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`). +### `all_repos.push.bitbucket_server_pull_request` +Pushes the branch to `origin` and then creates a Bitbucket pull request for the branch. +#### Required `push_settings` +- `base_url`: the Bitbucket server URL (eg `bitbucket.domain.com`) +- `username`: the Bitbucket username you will log in as. +- `app_password`: the authentication method for the above user to login with + - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage). + - We need the scope: Repositories -> Read +### `all_repos.push.gitlab_pull_request` +Pushes the branch to `origin` and then creates a GitLab pull request for the branch. +#### Required `push_settings` +- `base_url`: the GitLab server URL (eg `https://{gitlab.domain.com}/api/v4`) +- `api_key`: the api key which the user will log in as. + - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a + personal access token. + - We need the scope: `write_repository`. +- `api_key_env`: alternatively API key can also be passed via an environment variable +### `all_repos.push.readonly` +Does nothing. +#### `push_settings` +There are no configurable settings for `readonly`. +## Writing your own push module +First create a module. This module must have the following api: +### A `Settings` class +This class will receive keyword arguments for all values in the `push_settings` +dictionary. +### `def push(settings: Settings, branch_name: str) -> None:` +This callable will be passed an instance of your `Settings` class. It should +deploy the branch. The function will be called with the root of the +repository as the `cwd`. +## Writing an autofixer +An autofixer applies a change over all repositories. +`all-repos` provides several api functions to write your autofixers with: +### `all_repos.autofix_lib.add_fixer_args` +```python +def add_fixer_args(parser): +``` +Adds the autofixer cli options. +Options: +- `--dry-run`: show what would happen but do not push. +- `-i` / `--interactive`: interactively approve / deny fixes. +- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete + the operation. Specify 0 or -1 to match the number of cpus. (default `1`). +- `--limit LIMIT`: maximum number of repos to process (default: unlimited). +- `--author AUTHOR`: override commit author. This is passed directly to + `git commit`. An example: `--author='Herp Derp <herp.derp@umich.edu>'`. +- `--repos [REPOS [REPOS ...]]`: run against specific repositories instead. + This is especially useful with `xargs autofixer ... --repos`. This can be + used to specify repositories which are not managed by `all-repos`. +### `all_repos.autofix_lib.from_cli` +```python +def from_cli(args, *, find_repos, msg, branch_name): +``` +Parse cli arguments and produce `autofix_lib` primitives. Returns +`(repos, config, commit, autofix_settings)`. This is handled separately from +`fix` to allow for fixers to adjust arguments. +- `find_repos`: callback taking `Config` as a positional argument. +- `msg`: commit message. +- `branch_name`: identifier used to construct the branch name. +### `all_repos.autofix_lib.fix` +```python +def fix( + repos, *, + apply_fix, + check_fix=_noop_check_fix, + config: Config, + commit: Commit, + autofix_settings: AutofixSettings, +): +``` +Apply the fix. +- `apply_fix`: callback which will be called once per repository. The `cwd` + when the function is called will be the root of the repository. +### `all_repos.autofix_lib.run` +```python +def run(*cmd, **kwargs): +``` +Wrapper around `subprocess.run` which prints the command it will run. Unlike +`subprocess.run`, this defaults `check=True` unless explicitly disabled. +### Example autofixer +The trivial autofixer is as follows: +```python +import argparse +from all_repos import autofix_lib +def find_repos(config): + return [] +def apply_fix(): + pass +def main(argv=None): + parser = argparse.ArgumentParser() + autofix_lib.add_fixer_args(parser) + args = parser.parse_args(argv) + repos, config, commit, autofix_settings = autofix_lib.from_cli( + args, find_repos=find_repos, msg='msg', branch_name='branch-name', + ) + autofix_lib.fix( + repos, apply_fix=apply_fix, config=config, commit=commit, + autofix_settings=autofix_settings, + ) +if __name__ == '__main__': + raise SystemExit(main()) +``` +You can find some more involved examples in [all_repos/autofix](https://github.com/asottile/all-repos/tree/main/all_repos/autofix): +- `all_repos.autofix.azure_pipelines_autoupdate`: upgrade pinned azure + pipelines template repository references. +- `all_repos.autofix.pre_commit_autoupdate`: runs `pre-commit autoupdate`. +- `all_repos.autofix.pre_commit_autopep8_migrate`: migrates `autopep8-wrapper` + from [pre-commit/pre-commit-hooks] to [mirrors-autopep8]. +- `all_repos.autofix.pre_commit_cache_dir`: updates the cache directory + for travis-ci / appveyor for pre-commit 1.x. +- `all_repos.autofix.pre_commit_flake8_migrate`: migrates `flake8` from + [pre-commit/pre-commit-hooks] to [pycqa/flake8]. +- `all_repos.autofix.pre_commit_migrate_config`: runs + `pre-commit migrate-config`. +- `all_repos.autofix.setup_py_upgrade`: runs [setup-py-upgrade] and then + [setup-cfg-fmt] to migrate `setup.py` to `setup.cfg`. +[pre-commit/pre-commit-hooks]: https://github.com/pre-commit/pre-commit-hooks +[mirrors-autopep8]: https://github.com/pre-commit/mirrors-autopep8 +[pycqa/flake8]: https://gitlab.com/pycqa/flake8 +[setup-py-upgrade]: https://github.com/asottile/setup-py-upgrade +[setup-cfg-fmt]: https://github.com/asottile/setup-cfg-fmt + +%prep +%autosetup -n all-repos-1.25.0 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-all-repos -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Mon May 15 2023 Python_Bot <Python_Bot@openeuler.org> - 1.25.0-1 +- Package Spec generated |
