diff options
| author | CoprDistGit <infra@openeuler.org> | 2023-04-10 17:57:25 +0000 |
|---|---|---|
| committer | CoprDistGit <infra@openeuler.org> | 2023-04-10 17:57:25 +0000 |
| commit | 6816e0b20f5b0eb5ad2e8286ee41e567820a0a4f (patch) | |
| tree | 9f96f3d3b1f81c9a95275aa964da60e11b583c82 /python-jupyterhub-idle-culler.spec | |
| parent | 589141080526ba5ca13b53ba762ba152ab8c04fe (diff) | |
automatic import of python-jupyterhub-idle-culler
Diffstat (limited to 'python-jupyterhub-idle-culler.spec')
| -rw-r--r-- | python-jupyterhub-idle-culler.spec | 920 |
1 files changed, 920 insertions, 0 deletions
diff --git a/python-jupyterhub-idle-culler.spec b/python-jupyterhub-idle-culler.spec new file mode 100644 index 0000000..3a9f1db --- /dev/null +++ b/python-jupyterhub-idle-culler.spec @@ -0,0 +1,920 @@ +%global _empty_manifest_terminate_build 0 +Name: python-jupyterhub-idle-culler +Version: 1.2.1 +Release: 1 +Summary: please add a summary manually as the author left a blank one +License: 3-BSD +URL: https://jupyter.org +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/e8/be/f225c5d4cd186f793e46ababecee6ee62b2a57906a0ec1fc0453cda6c6ed/jupyterhub-idle-culler-1.2.1.tar.gz +BuildArch: noarch + +Requires: python3-tornado +Requires: python3-dateutil + +%description +# JupyterHub Idle Culler Service + +[](https://github.com/jupyterhub/jupyterhub-idle-culler/actions) +[](https://pypi.python.org/pypi/jupyterhub-idle-culler) +[](https://github.com/jupyterhub/jupyterhub-idle-culler/issues) +[](https://discourse.jupyter.org/c/jupyterhub) +[](https://gitter.im/jupyterhub/jupyterhub) + +`jupyterhub-idle-culler` provides a JupyterHub service to identify and shut down idle or long-running Jupyter Notebook servers. +The exact actions performed are dependent on the used spawner for the Jupyter Notebook server (e.g. the default [LocalProcessSpawner](https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#localprocessspawner>), [kubespawner](https://github.com/jupyterhub/kubespawner), or [dockerspawner](https://github.com/jupyterhub/dockerspawner)). +In addition, if explicitly requested, all users whose Jupyter Notebook servers have been shut down this way are deleted as JupyterHub users from the internal database. This neither affects the authentication method which continues to allow those users to log in nor does it delete persisted user data (e.g. stored in docker volumes for dockerspawner or in persisted volumes for kubespawner). + +## Setup + +### Installation + +```bash +pip install jupyterhub-idle-culler +``` + +### Permissions + +Prior to JupyterHub 2.0, the `jupyterhub-idle-culler` required full administrative privileges, +in order to have sufficient permissions to stop servers on behalf of users. + +JupyterHub 2.0 introduces [scopes][] to allow for more fine-grained permission control. +This means that the configured culler service does not need full administrative privileges anymore. +It can be assigned only the permissions it needs. + +[scopes]: https://jupyterhub.readthedocs.io/en/latest/rbac/scopes.html#available-scopes + +`jupyterhub-idle-culler` requires the following scopes to function: + +- `list:users` - access to the user list API, our source of information about who to cull +- `read:users:activity` - read the last_activity field of the user +- `delete:servers` - management of servers (this includes stopping servers) +- `admin:users` (**optional**) - only needed if using `--cull-users` + +To assign the service the appropriate permissions, declare a role in your `jupyterhub_config.py`: + +```python +c.JupyterHub.load_roles = [ + { + "name": "jupyterhub-idle-culler-role", + "scopes": [ + "list:users", + "read:users:activity", + "delete:servers", + # "admin:users", # if using --cull-users + ], + # assignment of role's permissions to: + "services": ["jupyterhub-idle-culler-service"], + } +] +``` + +### As a hub managed service + +In `jupyterhub_config.py`, add the following dictionary for the idle-culler +Service to the `c.JupyterHub.services` list: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "command": [ + sys.executable, + "-m", "jupyterhub_idle_culler", + "--timeout=3600", + ], + # "admin": True, + } +] +``` + +where: + +- `"command"` indicates that the Service will be managed by the Hub, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +### As a standalone script + +`jupyterhub-idle-culler` can also be run as a standalone script. It can +access the hub's api with a service token. + +Register the service token with JupyterHub in `jupyterhub_config.py`: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "api_token": "...", + # "admin": True, + } +] +``` + +where: + +- `"api_token"` contains a secret token, e.g. generated by `openssl rand -hex 32`, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +and store the same token in a `JUPYTERHUB_API_TOKEN` environment variable. +Then start `jupyterhub-idle-culler` manually. + +```bash +export JUPYTERHUB_API_TOKEN=api_token_above... +python3 -m jupyterhub-idle-culler [--timeout=900] [--url=http://localhost:8081/hub/api] +``` + +## Command line flags + +``` + --api-page-size Number of users to request per page, when + using JupyterHub 2.0's paginated user list + API. Default: user the server-side default + configured page size. (default 0) + --concurrency Limit the number of concurrent requests made + to the Hub. Deleting a lot of users at the + same time can slow down the Hub, so limit + the number of API requests we have + outstanding at any given time. (default 10) + --cull-admin-users Whether admin users should be culled (only + if --cull-users=true). (default True) + --cull-every The interval (in seconds) for checking for + idle servers to cull. (default 0) + --cull-users Cull users in addition to servers. This is + for use in temporary-user cases such as + tmpnb. (default False) + --internal-certs-location The location of generated internal-ssl + certificates (only needed with --ssl- + enabled=true). (default internal-ssl) + --max-age The maximum age (in seconds) of servers that + should be culled even if they are active. + (default 0) + --remove-named-servers Remove named servers in addition to stopping + them. This is useful for a BinderHub that + uses authentication and named servers. + (default False) + --ssl-enabled Whether the Jupyter API endpoint has TLS + enabled. (default False) + --timeout The idle timeout (in seconds). (default 600) + --url The JupyterHub API URL. +``` + +## Caveats + +1. last_activity is not updated with high frequency, so cull timeout should be + greater than the sum of: + + - single-user websocket ping interval (default: 30 seconds) + - `JupyterHub.last_activity_interval` (default: 5 minutes) + +2. The same `--timeout` and `--max-age` values are used to cull + users and users' servers. If you want a different value for users and servers, + you should add this script to the services list twice, just with different + `name`s, different values, and one with the `--cull-users` option. + +3. By default HTTP requests to the hub timeout after 60 seconds. This can be + changed by setting the `JUPYTERHUB_REQUEST_TIMEOUT` environment variable. + +## How it works + +jupytehrub-idle-culler lists available users via JupyterHub's [/users][users-api] REST API. + +[users-api]: https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#path--users + +jupyterhub-idle-culler culls user servers using JupyterHub's REST API +([/users/{name}/server](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--server-delete) +or +[/users/{name}/servers/{server_name}](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--servers--server_name--delete)), +and makes the culling decisions based on its configuration and what JupyterHub +reports about the user servers via its REST API +[(/users)][users-api] +where user servers' `last_activity` is reported back. + +The `last_activity` that JupyterHub reports is the most recent summary of +information updated at a regular interval via the [`update_last_activity` +function](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/app.py#L2646) +that combines two sources of information. + +1. **The proxy's routes data** + + The `update_last_activity` function will [ask the + proxy](https://jupyterhub.readthedocs.io/en/stable/reference/proxy.html#retrieving-routes) + for the active routes like `/user/user1` and collects associated + `last_activity` data if it is available. This activity represents + successfully proxies network traffic. + + `last_activity` data for routes will be available when using + [configurable-http-proxy](https://github.com/jupyterhub/configurable-http-proxy#readme) + as JupyterHub does by default, but if for example + [traefik-proxy](https://github.com/jupyterhub/traefik-proxy#readme) is used + as it is in the [TLJH distribution](https://tljh.jupyter.org), no such data + will be available. + +2. **The user server's activity reports** + + The `update_last_activity` function also reads JupyterHub's database that + keeps state about servers `last_activity`. These database records are updated + whenever a server notifies JupyterHub about activity, as they are + responsible to do. + + Servers notify JupyterHub about activity by being started by the + [`jupyterhub-singleuser`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/setup.py#L115) + script that is made available by installing jupyterhub (or `jupyterhub-base` + on conda-forge). + + The `jupyterhub-singleuser` script launches a modified server application + that keeps JupyterHub updated with the server activity via the + [`notify_activity`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/singleuser/mixins.py#L497) + function. + + The `notify_activity` function in turn make use of the server applications + `last_activity` function (see implementation in + [NotebookApp](https://github.com/jupyter/notebook/blob/v6.4.0/notebook/notebookapp.py#L392-L397) + and + [ServerApp](https://github.com/jupyter-server/jupyter_server/blob/v1.9.0/jupyter_server/serverapp.py#L375) + respectively) that that combines information from API activity, kernel + activity, kernel shutdown, and terminal activity. This activity also covers + activity of applications like RStudio running via `jupyter-server-proxy`. + +Here is a summary of what's described so far: + +1. jupyterhub-idle-culler culls servers via JupyterHub's REST API. +2. jupyterhub-idle-culler makes decisions based on information retrieved by + JupyterHub REST API. +3. JupyterHub REST API reports information regularly updated by summarizing + information gained by: asking the proxy about routes' activity, and by + retaining activity information reported by the servers. + +Now, as the server's kernel activity influence the activity that servers will +notify JupyterHub about, the kernel activity in turn influences +jupyterhub-idle-culler. Due to this, it can be relevant to also learn a little +about a mechanism to _cull idle kernels_ as well even though +jupyterhub-idle-culler isn't involved in that. + +The default kernel manager, the MappingKernelManager, can be configured to cull +idle kernels. Its configuration is documented in +[NotebookApp's](https://jupyter-notebook.readthedocs.io/en/stable/config.html#options) +and +[ServerApp's](https://jupyter-server.readthedocs.io/en/latest/full-config.html) +respective documentation, and here are some relevant kernel culling +configuration options: + +- `MappingKernelManager.cull_busy` +- `MappingKernelManager.cull_idle_timeout` +- `MappingKernelManager.cull_interval` +- `MappingKernelManager.cull_connected` + + Note that `cull_connected` can be tricky to understand for JupyterLab as a + browser having a web-socket connection to a kernel or not isn't as obvious as + it was in the classical Jupyter notebook UI. See [this issue for more + details](https://github.com/jupyterlab/jupyterlab/issues/6893). + + Also note that configuration of MappingKernelManager should be made on the + user server itself, for example via a `jupyter_notebook_config.py` file in + `/etc/jupyter` or `/usr/local/etc/jupyter` rather than where JupyterHub is + running. + +Finally, note that a Jupyter Notebook server can shut itself down without +without intervention by jupyterhub-idle-culler if +`NotebookApp.shutdown_no_activity_timeout` is configured. + +### Caveats + +#### Pagination + +JupyterHub 2.0 introduces pagination to the [/users][users-api] API endpoint. +This pagination does not guarantee a consistent snapshot +for consecutive requests spread over time, +so it is possible for a highly active hub to occasionally miss culling users crossing page boundaries between requests. +This is expected to be an infrequent occurrence and only result in delaying a server being culled by one cull interval +in realistic scenarios, so of minor consequence in JupyterHub. + +The issue can be mitigated by requesting a larger page size, +via e.g. `--api-page-size=200`, +but feel free to open an issue if this is causing a problem for you. + + + + +%package -n python3-jupyterhub-idle-culler +Summary: please add a summary manually as the author left a blank one +Provides: python-jupyterhub-idle-culler +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-jupyterhub-idle-culler +# JupyterHub Idle Culler Service + +[](https://github.com/jupyterhub/jupyterhub-idle-culler/actions) +[](https://pypi.python.org/pypi/jupyterhub-idle-culler) +[](https://github.com/jupyterhub/jupyterhub-idle-culler/issues) +[](https://discourse.jupyter.org/c/jupyterhub) +[](https://gitter.im/jupyterhub/jupyterhub) + +`jupyterhub-idle-culler` provides a JupyterHub service to identify and shut down idle or long-running Jupyter Notebook servers. +The exact actions performed are dependent on the used spawner for the Jupyter Notebook server (e.g. the default [LocalProcessSpawner](https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#localprocessspawner>), [kubespawner](https://github.com/jupyterhub/kubespawner), or [dockerspawner](https://github.com/jupyterhub/dockerspawner)). +In addition, if explicitly requested, all users whose Jupyter Notebook servers have been shut down this way are deleted as JupyterHub users from the internal database. This neither affects the authentication method which continues to allow those users to log in nor does it delete persisted user data (e.g. stored in docker volumes for dockerspawner or in persisted volumes for kubespawner). + +## Setup + +### Installation + +```bash +pip install jupyterhub-idle-culler +``` + +### Permissions + +Prior to JupyterHub 2.0, the `jupyterhub-idle-culler` required full administrative privileges, +in order to have sufficient permissions to stop servers on behalf of users. + +JupyterHub 2.0 introduces [scopes][] to allow for more fine-grained permission control. +This means that the configured culler service does not need full administrative privileges anymore. +It can be assigned only the permissions it needs. + +[scopes]: https://jupyterhub.readthedocs.io/en/latest/rbac/scopes.html#available-scopes + +`jupyterhub-idle-culler` requires the following scopes to function: + +- `list:users` - access to the user list API, our source of information about who to cull +- `read:users:activity` - read the last_activity field of the user +- `delete:servers` - management of servers (this includes stopping servers) +- `admin:users` (**optional**) - only needed if using `--cull-users` + +To assign the service the appropriate permissions, declare a role in your `jupyterhub_config.py`: + +```python +c.JupyterHub.load_roles = [ + { + "name": "jupyterhub-idle-culler-role", + "scopes": [ + "list:users", + "read:users:activity", + "delete:servers", + # "admin:users", # if using --cull-users + ], + # assignment of role's permissions to: + "services": ["jupyterhub-idle-culler-service"], + } +] +``` + +### As a hub managed service + +In `jupyterhub_config.py`, add the following dictionary for the idle-culler +Service to the `c.JupyterHub.services` list: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "command": [ + sys.executable, + "-m", "jupyterhub_idle_culler", + "--timeout=3600", + ], + # "admin": True, + } +] +``` + +where: + +- `"command"` indicates that the Service will be managed by the Hub, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +### As a standalone script + +`jupyterhub-idle-culler` can also be run as a standalone script. It can +access the hub's api with a service token. + +Register the service token with JupyterHub in `jupyterhub_config.py`: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "api_token": "...", + # "admin": True, + } +] +``` + +where: + +- `"api_token"` contains a secret token, e.g. generated by `openssl rand -hex 32`, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +and store the same token in a `JUPYTERHUB_API_TOKEN` environment variable. +Then start `jupyterhub-idle-culler` manually. + +```bash +export JUPYTERHUB_API_TOKEN=api_token_above... +python3 -m jupyterhub-idle-culler [--timeout=900] [--url=http://localhost:8081/hub/api] +``` + +## Command line flags + +``` + --api-page-size Number of users to request per page, when + using JupyterHub 2.0's paginated user list + API. Default: user the server-side default + configured page size. (default 0) + --concurrency Limit the number of concurrent requests made + to the Hub. Deleting a lot of users at the + same time can slow down the Hub, so limit + the number of API requests we have + outstanding at any given time. (default 10) + --cull-admin-users Whether admin users should be culled (only + if --cull-users=true). (default True) + --cull-every The interval (in seconds) for checking for + idle servers to cull. (default 0) + --cull-users Cull users in addition to servers. This is + for use in temporary-user cases such as + tmpnb. (default False) + --internal-certs-location The location of generated internal-ssl + certificates (only needed with --ssl- + enabled=true). (default internal-ssl) + --max-age The maximum age (in seconds) of servers that + should be culled even if they are active. + (default 0) + --remove-named-servers Remove named servers in addition to stopping + them. This is useful for a BinderHub that + uses authentication and named servers. + (default False) + --ssl-enabled Whether the Jupyter API endpoint has TLS + enabled. (default False) + --timeout The idle timeout (in seconds). (default 600) + --url The JupyterHub API URL. +``` + +## Caveats + +1. last_activity is not updated with high frequency, so cull timeout should be + greater than the sum of: + + - single-user websocket ping interval (default: 30 seconds) + - `JupyterHub.last_activity_interval` (default: 5 minutes) + +2. The same `--timeout` and `--max-age` values are used to cull + users and users' servers. If you want a different value for users and servers, + you should add this script to the services list twice, just with different + `name`s, different values, and one with the `--cull-users` option. + +3. By default HTTP requests to the hub timeout after 60 seconds. This can be + changed by setting the `JUPYTERHUB_REQUEST_TIMEOUT` environment variable. + +## How it works + +jupytehrub-idle-culler lists available users via JupyterHub's [/users][users-api] REST API. + +[users-api]: https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#path--users + +jupyterhub-idle-culler culls user servers using JupyterHub's REST API +([/users/{name}/server](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--server-delete) +or +[/users/{name}/servers/{server_name}](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--servers--server_name--delete)), +and makes the culling decisions based on its configuration and what JupyterHub +reports about the user servers via its REST API +[(/users)][users-api] +where user servers' `last_activity` is reported back. + +The `last_activity` that JupyterHub reports is the most recent summary of +information updated at a regular interval via the [`update_last_activity` +function](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/app.py#L2646) +that combines two sources of information. + +1. **The proxy's routes data** + + The `update_last_activity` function will [ask the + proxy](https://jupyterhub.readthedocs.io/en/stable/reference/proxy.html#retrieving-routes) + for the active routes like `/user/user1` and collects associated + `last_activity` data if it is available. This activity represents + successfully proxies network traffic. + + `last_activity` data for routes will be available when using + [configurable-http-proxy](https://github.com/jupyterhub/configurable-http-proxy#readme) + as JupyterHub does by default, but if for example + [traefik-proxy](https://github.com/jupyterhub/traefik-proxy#readme) is used + as it is in the [TLJH distribution](https://tljh.jupyter.org), no such data + will be available. + +2. **The user server's activity reports** + + The `update_last_activity` function also reads JupyterHub's database that + keeps state about servers `last_activity`. These database records are updated + whenever a server notifies JupyterHub about activity, as they are + responsible to do. + + Servers notify JupyterHub about activity by being started by the + [`jupyterhub-singleuser`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/setup.py#L115) + script that is made available by installing jupyterhub (or `jupyterhub-base` + on conda-forge). + + The `jupyterhub-singleuser` script launches a modified server application + that keeps JupyterHub updated with the server activity via the + [`notify_activity`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/singleuser/mixins.py#L497) + function. + + The `notify_activity` function in turn make use of the server applications + `last_activity` function (see implementation in + [NotebookApp](https://github.com/jupyter/notebook/blob/v6.4.0/notebook/notebookapp.py#L392-L397) + and + [ServerApp](https://github.com/jupyter-server/jupyter_server/blob/v1.9.0/jupyter_server/serverapp.py#L375) + respectively) that that combines information from API activity, kernel + activity, kernel shutdown, and terminal activity. This activity also covers + activity of applications like RStudio running via `jupyter-server-proxy`. + +Here is a summary of what's described so far: + +1. jupyterhub-idle-culler culls servers via JupyterHub's REST API. +2. jupyterhub-idle-culler makes decisions based on information retrieved by + JupyterHub REST API. +3. JupyterHub REST API reports information regularly updated by summarizing + information gained by: asking the proxy about routes' activity, and by + retaining activity information reported by the servers. + +Now, as the server's kernel activity influence the activity that servers will +notify JupyterHub about, the kernel activity in turn influences +jupyterhub-idle-culler. Due to this, it can be relevant to also learn a little +about a mechanism to _cull idle kernels_ as well even though +jupyterhub-idle-culler isn't involved in that. + +The default kernel manager, the MappingKernelManager, can be configured to cull +idle kernels. Its configuration is documented in +[NotebookApp's](https://jupyter-notebook.readthedocs.io/en/stable/config.html#options) +and +[ServerApp's](https://jupyter-server.readthedocs.io/en/latest/full-config.html) +respective documentation, and here are some relevant kernel culling +configuration options: + +- `MappingKernelManager.cull_busy` +- `MappingKernelManager.cull_idle_timeout` +- `MappingKernelManager.cull_interval` +- `MappingKernelManager.cull_connected` + + Note that `cull_connected` can be tricky to understand for JupyterLab as a + browser having a web-socket connection to a kernel or not isn't as obvious as + it was in the classical Jupyter notebook UI. See [this issue for more + details](https://github.com/jupyterlab/jupyterlab/issues/6893). + + Also note that configuration of MappingKernelManager should be made on the + user server itself, for example via a `jupyter_notebook_config.py` file in + `/etc/jupyter` or `/usr/local/etc/jupyter` rather than where JupyterHub is + running. + +Finally, note that a Jupyter Notebook server can shut itself down without +without intervention by jupyterhub-idle-culler if +`NotebookApp.shutdown_no_activity_timeout` is configured. + +### Caveats + +#### Pagination + +JupyterHub 2.0 introduces pagination to the [/users][users-api] API endpoint. +This pagination does not guarantee a consistent snapshot +for consecutive requests spread over time, +so it is possible for a highly active hub to occasionally miss culling users crossing page boundaries between requests. +This is expected to be an infrequent occurrence and only result in delaying a server being culled by one cull interval +in realistic scenarios, so of minor consequence in JupyterHub. + +The issue can be mitigated by requesting a larger page size, +via e.g. `--api-page-size=200`, +but feel free to open an issue if this is causing a problem for you. + + + + +%package help +Summary: Development documents and examples for jupyterhub-idle-culler +Provides: python3-jupyterhub-idle-culler-doc +%description help +# JupyterHub Idle Culler Service + +[](https://github.com/jupyterhub/jupyterhub-idle-culler/actions) +[](https://pypi.python.org/pypi/jupyterhub-idle-culler) +[](https://github.com/jupyterhub/jupyterhub-idle-culler/issues) +[](https://discourse.jupyter.org/c/jupyterhub) +[](https://gitter.im/jupyterhub/jupyterhub) + +`jupyterhub-idle-culler` provides a JupyterHub service to identify and shut down idle or long-running Jupyter Notebook servers. +The exact actions performed are dependent on the used spawner for the Jupyter Notebook server (e.g. the default [LocalProcessSpawner](https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#localprocessspawner>), [kubespawner](https://github.com/jupyterhub/kubespawner), or [dockerspawner](https://github.com/jupyterhub/dockerspawner)). +In addition, if explicitly requested, all users whose Jupyter Notebook servers have been shut down this way are deleted as JupyterHub users from the internal database. This neither affects the authentication method which continues to allow those users to log in nor does it delete persisted user data (e.g. stored in docker volumes for dockerspawner or in persisted volumes for kubespawner). + +## Setup + +### Installation + +```bash +pip install jupyterhub-idle-culler +``` + +### Permissions + +Prior to JupyterHub 2.0, the `jupyterhub-idle-culler` required full administrative privileges, +in order to have sufficient permissions to stop servers on behalf of users. + +JupyterHub 2.0 introduces [scopes][] to allow for more fine-grained permission control. +This means that the configured culler service does not need full administrative privileges anymore. +It can be assigned only the permissions it needs. + +[scopes]: https://jupyterhub.readthedocs.io/en/latest/rbac/scopes.html#available-scopes + +`jupyterhub-idle-culler` requires the following scopes to function: + +- `list:users` - access to the user list API, our source of information about who to cull +- `read:users:activity` - read the last_activity field of the user +- `delete:servers` - management of servers (this includes stopping servers) +- `admin:users` (**optional**) - only needed if using `--cull-users` + +To assign the service the appropriate permissions, declare a role in your `jupyterhub_config.py`: + +```python +c.JupyterHub.load_roles = [ + { + "name": "jupyterhub-idle-culler-role", + "scopes": [ + "list:users", + "read:users:activity", + "delete:servers", + # "admin:users", # if using --cull-users + ], + # assignment of role's permissions to: + "services": ["jupyterhub-idle-culler-service"], + } +] +``` + +### As a hub managed service + +In `jupyterhub_config.py`, add the following dictionary for the idle-culler +Service to the `c.JupyterHub.services` list: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "command": [ + sys.executable, + "-m", "jupyterhub_idle_culler", + "--timeout=3600", + ], + # "admin": True, + } +] +``` + +where: + +- `"command"` indicates that the Service will be managed by the Hub, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +### As a standalone script + +`jupyterhub-idle-culler` can also be run as a standalone script. It can +access the hub's api with a service token. + +Register the service token with JupyterHub in `jupyterhub_config.py`: + +```python +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "api_token": "...", + # "admin": True, + } +] +``` + +where: + +- `"api_token"` contains a secret token, e.g. generated by `openssl rand -hex 32`, and +- `"admin": True` grants admin permissions to this Service and is only meant for + use with jupyterhub < 2.0; see [above][permissions]. + +and store the same token in a `JUPYTERHUB_API_TOKEN` environment variable. +Then start `jupyterhub-idle-culler` manually. + +```bash +export JUPYTERHUB_API_TOKEN=api_token_above... +python3 -m jupyterhub-idle-culler [--timeout=900] [--url=http://localhost:8081/hub/api] +``` + +## Command line flags + +``` + --api-page-size Number of users to request per page, when + using JupyterHub 2.0's paginated user list + API. Default: user the server-side default + configured page size. (default 0) + --concurrency Limit the number of concurrent requests made + to the Hub. Deleting a lot of users at the + same time can slow down the Hub, so limit + the number of API requests we have + outstanding at any given time. (default 10) + --cull-admin-users Whether admin users should be culled (only + if --cull-users=true). (default True) + --cull-every The interval (in seconds) for checking for + idle servers to cull. (default 0) + --cull-users Cull users in addition to servers. This is + for use in temporary-user cases such as + tmpnb. (default False) + --internal-certs-location The location of generated internal-ssl + certificates (only needed with --ssl- + enabled=true). (default internal-ssl) + --max-age The maximum age (in seconds) of servers that + should be culled even if they are active. + (default 0) + --remove-named-servers Remove named servers in addition to stopping + them. This is useful for a BinderHub that + uses authentication and named servers. + (default False) + --ssl-enabled Whether the Jupyter API endpoint has TLS + enabled. (default False) + --timeout The idle timeout (in seconds). (default 600) + --url The JupyterHub API URL. +``` + +## Caveats + +1. last_activity is not updated with high frequency, so cull timeout should be + greater than the sum of: + + - single-user websocket ping interval (default: 30 seconds) + - `JupyterHub.last_activity_interval` (default: 5 minutes) + +2. The same `--timeout` and `--max-age` values are used to cull + users and users' servers. If you want a different value for users and servers, + you should add this script to the services list twice, just with different + `name`s, different values, and one with the `--cull-users` option. + +3. By default HTTP requests to the hub timeout after 60 seconds. This can be + changed by setting the `JUPYTERHUB_REQUEST_TIMEOUT` environment variable. + +## How it works + +jupytehrub-idle-culler lists available users via JupyterHub's [/users][users-api] REST API. + +[users-api]: https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#path--users + +jupyterhub-idle-culler culls user servers using JupyterHub's REST API +([/users/{name}/server](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--server-delete) +or +[/users/{name}/servers/{server_name}](https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html#operation--users--name--servers--server_name--delete)), +and makes the culling decisions based on its configuration and what JupyterHub +reports about the user servers via its REST API +[(/users)][users-api] +where user servers' `last_activity` is reported back. + +The `last_activity` that JupyterHub reports is the most recent summary of +information updated at a regular interval via the [`update_last_activity` +function](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/app.py#L2646) +that combines two sources of information. + +1. **The proxy's routes data** + + The `update_last_activity` function will [ask the + proxy](https://jupyterhub.readthedocs.io/en/stable/reference/proxy.html#retrieving-routes) + for the active routes like `/user/user1` and collects associated + `last_activity` data if it is available. This activity represents + successfully proxies network traffic. + + `last_activity` data for routes will be available when using + [configurable-http-proxy](https://github.com/jupyterhub/configurable-http-proxy#readme) + as JupyterHub does by default, but if for example + [traefik-proxy](https://github.com/jupyterhub/traefik-proxy#readme) is used + as it is in the [TLJH distribution](https://tljh.jupyter.org), no such data + will be available. + +2. **The user server's activity reports** + + The `update_last_activity` function also reads JupyterHub's database that + keeps state about servers `last_activity`. These database records are updated + whenever a server notifies JupyterHub about activity, as they are + responsible to do. + + Servers notify JupyterHub about activity by being started by the + [`jupyterhub-singleuser`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/setup.py#L115) + script that is made available by installing jupyterhub (or `jupyterhub-base` + on conda-forge). + + The `jupyterhub-singleuser` script launches a modified server application + that keeps JupyterHub updated with the server activity via the + [`notify_activity`](https://github.com/jupyterhub/jupyterhub/blob/1.4.2/jupyterhub/singleuser/mixins.py#L497) + function. + + The `notify_activity` function in turn make use of the server applications + `last_activity` function (see implementation in + [NotebookApp](https://github.com/jupyter/notebook/blob/v6.4.0/notebook/notebookapp.py#L392-L397) + and + [ServerApp](https://github.com/jupyter-server/jupyter_server/blob/v1.9.0/jupyter_server/serverapp.py#L375) + respectively) that that combines information from API activity, kernel + activity, kernel shutdown, and terminal activity. This activity also covers + activity of applications like RStudio running via `jupyter-server-proxy`. + +Here is a summary of what's described so far: + +1. jupyterhub-idle-culler culls servers via JupyterHub's REST API. +2. jupyterhub-idle-culler makes decisions based on information retrieved by + JupyterHub REST API. +3. JupyterHub REST API reports information regularly updated by summarizing + information gained by: asking the proxy about routes' activity, and by + retaining activity information reported by the servers. + +Now, as the server's kernel activity influence the activity that servers will +notify JupyterHub about, the kernel activity in turn influences +jupyterhub-idle-culler. Due to this, it can be relevant to also learn a little +about a mechanism to _cull idle kernels_ as well even though +jupyterhub-idle-culler isn't involved in that. + +The default kernel manager, the MappingKernelManager, can be configured to cull +idle kernels. Its configuration is documented in +[NotebookApp's](https://jupyter-notebook.readthedocs.io/en/stable/config.html#options) +and +[ServerApp's](https://jupyter-server.readthedocs.io/en/latest/full-config.html) +respective documentation, and here are some relevant kernel culling +configuration options: + +- `MappingKernelManager.cull_busy` +- `MappingKernelManager.cull_idle_timeout` +- `MappingKernelManager.cull_interval` +- `MappingKernelManager.cull_connected` + + Note that `cull_connected` can be tricky to understand for JupyterLab as a + browser having a web-socket connection to a kernel or not isn't as obvious as + it was in the classical Jupyter notebook UI. See [this issue for more + details](https://github.com/jupyterlab/jupyterlab/issues/6893). + + Also note that configuration of MappingKernelManager should be made on the + user server itself, for example via a `jupyter_notebook_config.py` file in + `/etc/jupyter` or `/usr/local/etc/jupyter` rather than where JupyterHub is + running. + +Finally, note that a Jupyter Notebook server can shut itself down without +without intervention by jupyterhub-idle-culler if +`NotebookApp.shutdown_no_activity_timeout` is configured. + +### Caveats + +#### Pagination + +JupyterHub 2.0 introduces pagination to the [/users][users-api] API endpoint. +This pagination does not guarantee a consistent snapshot +for consecutive requests spread over time, +so it is possible for a highly active hub to occasionally miss culling users crossing page boundaries between requests. +This is expected to be an infrequent occurrence and only result in delaying a server being culled by one cull interval +in realistic scenarios, so of minor consequence in JupyterHub. + +The issue can be mitigated by requesting a larger page size, +via e.g. `--api-page-size=200`, +but feel free to open an issue if this is causing a problem for you. + + + + +%prep +%autosetup -n jupyterhub-idle-culler-1.2.1 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-jupyterhub-idle-culler -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Mon Apr 10 2023 Python_Bot <Python_Bot@openeuler.org> - 1.2.1-1 +- Package Spec generated |
