diff --git a/.goreleaser.yaml b/.goreleaser.yaml
index d3f0e03..5049417 100644
--- a/.goreleaser.yaml
+++ b/.goreleaser.yaml
@@ -31,7 +31,6 @@ builds:
archives:
- files:
- mount.yaml
- rlcp: true
name_template: >-
{{ .ProjectName }}_{{ .Version }}_
{{- if eq .Os "darwin" }}macOS
@@ -41,7 +40,6 @@ archives:
files:
- mount.yaml
format: binary
- rlcp: true
name_template: >-
{{ .ProjectName }}_{{ .Version }}_
{{- if eq .Os "darwin" }}macOS
diff --git a/README.cn.md b/README.cn.md
index d04e077..64e8fa5 100644
--- a/README.cn.md
+++ b/README.cn.md
@@ -4,34 +4,25 @@
[English](README.md)
-## 沙箱服务
-
快速,简单,安全
-### 运行要求
+## 快速上手
-- Linux 内核版本 >= 3.10
-- 系统 Cgroup 文件系统挂载于 `/sys/fs/cgroup`(Systemd 默认)
+### 安装和运行
-### 系统架构
+下载对应平台预编译二进制文件 `executorserver` [Release](https://github.com/criyle/go-judge/releases) 并在终端开启
-```text
-+----------------------------------------------------------------------------+
-| 传输层 (HTTP / WebSocket / FFI / ...) |
-+----------------------------------------------------------------------------+
-| 工作协程 (运行环境池 和 运行环境生产者 ) |
-+-----------------------------------------------------------+----------------+
-| 运行环境 | 文件存储 |
-+--------------------+----------------+---------------------+----------+-----+
-| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | 共享内存 | 磁盘 |
-+--------------------+----------------+---------------------+----------+-----+
+或者使用 docker
+
+```bash
+docker run -it --rm --privileged --shm-size=256m -p 5050:5050 --name=executorserver criyle/executorserver
```
### REST API 接口
-沙箱服务提供 REST API 接口来在受限制的环境中运行程序。本质是 `envexec` 的简单封装。
+沙箱服务提供 REST API 接口来在受限制的环境中运行程序(默认监听于 `localhost:5050`)。
-- /run POST 在受限制的环境中运行程序(下面有例子)
+- **/run POST 在受限制的环境中运行程序(下面有例子)**
- /file GET 得到所有在文件存储中的文件 ID 到原始命名映射
- /file POST 上传一个文件到文件存储,返回一个文件 ID 用于提供给 /run 接口
- /file/:fileId GET 下载文件 ID 指定的文件
@@ -40,177 +31,6 @@
- /version 得到本程序编译版本和 go 语言运行时版本
- /config 得到本程序部分运行参数,包括沙箱详细参数
-监控端口(默认监听 `:5052`)
-
-- /metrics 提供 prometheus 版监控 (使用 `ES_ENABLE_METRICS=1` 环境变量开启)
-- /debug 提供 go 语言调试接口 (使用 `ES_ENABLE_DEBUG=1` 环境变量开启)
-
-### 命令行参数
-
-服务相关:
-
-- 默认监听地址是 `:5050`,使用 `-http-addr` 指定
-- 默认 gRPC 接口处于关闭状态,使用 `-enable-grpc` 开启
-- 默认 gRPC 监听地址是 `:5051` ,使用 `-grpc-addr` 指定
-- 默认日志等级是 info ,使用 `-silent` 关闭 或 使用 `-release` 开启 release 级别日志
-- 默认没有开启鉴权,使用 `-auth-token` 指定令牌鉴权
-- 默认没有开启 go 语言调试接口,使用 `-enable-debug` 开启,同时将日志层级设为 Debug
-- 默认没有开启 prometheus 监控接口,使用 `-enable-metrics` 开启
-- 在启用 go 语言调试接口或者 prometheus 监控接口的情况下,默认监控接口为 `:5052`,使用 `-monitor-addr` 指定
-
-沙箱相关:
-
-- 默认同时运行任务数为和 CPU 数量相同,使用 `-parallelism` 指定
-- 默认文件存储在内存里,使用 `-dir` 指定本地目录为文件存储
-- 默认 cgroup 的前缀为 `executor_server` ,使用 `-cgroup-prefix` 指定
-- 默认没有磁盘文件复制限制,使用 `-src-prefix` 限制 copyIn 操作文件目录前缀,使用逗号 `,` 分隔(需要绝对路径)(例如:`/bin,/usr`)
-- 默认时间和内存使用检查周期为 100 毫秒(`100ms`),使用 `-time-limit-checker-interval` 指定
-- 默认最大输出限制为 `256MiB`,使用 `-output-limit` 指定
-- 默认最大打开文件描述符为 `256`,使用 `-open-file-limit` 指定
-- 默认最大额外内存使用为 `16KiB` ,使用 `-extra-memory-limit` 指定
-- 默认最大 `copyOut` 文件大小为 `64MiB` ,使用 `-copy-out-limit` 指定
-- 使用 `-cpuset` 指定 `cpuset.cpus` (仅 Linux)
-- 默认容器用户开始区间为 10000 使用 `-container-cred-start` 指定(仅 Linux)
- - 举例,默认情况下第 0 个容器使用 10001 作为容器用户。第 1 个容器使用 10002 作为容器用户,以此类推
-- 使用 `-enable-cpu-rate` 开启 `cpu` cgroup 来启用 `cpuRate` 控制(仅 Linux)
- - 使用 `-cpu-cfs-period` 指定 cfs_period if cpu rate is enabled (default 100ms) (valid value: \[1ms, 1s\])
-- 使用 `-seccomp-conf` 指定 `seecomp` 过滤器(需要编译标志 `seccomp`,默认不开启)(仅 Linux)
-- 使用 `-pre-fork` 指定启动时创建的容器数量
-- 使用 `-tmp-fs-param` 指定容器内 `tmpfs` 的挂载参数(仅 Linux)
-- 使用 `-file-timeout` 指定文件存储文件最大时间。超出时间的文件将会删除。(举例 `30m`)
-- 使用 `-mount-conf` 指定沙箱文件系统挂载细节,详细请参见 `mount.yaml` (仅 Linux)
-- 使用 `-container-init-path` 指定 `cinit` 路径 (请不要使用,仅 debug) (仅 Linux)
-
-### 环境变量
-
-所有命令行参数都可以通过环境变量的形式来指定,(类似 `ES_HTTP_ADDR` 来指定 `-http-addr`)。使用 `executorserver --help` 查看所有环境变量
-
-### 安装和运行
-
-下载预编译二进制文件 [Release](https://github.com/criyle/go-judge/releases) 并在终端开启
-
-或者使用 docker
-
-```bash
-docker run -it --rm --privileged --shm-size=256m -p 5050:5050 criyle/executorserver
-```
-
-#### 编译 docker
-
-终端中运行 `docker build -t executorserver -f Dockerfile.exec .`
-
-沙箱服务需要特权级别 docker 来创建子容器和提供 cgroup 资源限制。
-
-### 编译沙箱终端
-
-编译 `go build ./cmd/executorshell`
-
-运行 `./executorshell`,需要打开 gRPC 接口来使用。提供一个沙箱内的终端环境。
-
-### /run 接口返回状态
-
-- Accepted: 程序在资源限制内正常退出
-- Memory Limit Exceeded: 超出内存限制
-- Time Limit Exceeded:
- - 超出 `timeLimit` 时间限制
- - 或者超过 `clockLimit` 等待时间限制
-- Output Limit Exceeded:
- - 超出 `pipeCollector` 限制
- - 或者超出 `-output-limit` 最大输出限制
-- File Error:
- - `copyIn` 指定文件不存在
- - 或者 `copyIn` 指定文件大小超出沙箱文件系统限制
- - 或者 `copyOut` 指定文件不存在
-- Non Zero Exit Status: 程序用非 0 返回值退出
-- Signalled: 程序收到结束信号而退出(例如 `SIGSEGV`)
-- Dangerous Syscall: 程序被 `seccomp` 过滤器结束
-- Internal Error:
- - 指定程序路径不存在
- - 或者容器创建失败
- - 比如使用非特权 docker
- - 或者在个人目录下以 root 权限运行
- - 或者其他错误
-
-### 容器的文件系统
-
-在 Linux 平台,默认只读挂载点包括主机的 `/lib`, `/lib64`, `/usr`, `/bin`, `/etc/ld.so.cache`, `/etc/alternatives`, `/etc/fpc.cfg`, `/dev/null`, `/dev/urandom`, `/dev/random`, `/dev/zero`, `/dev/full` 和临时文件系统 `/w`, `/tmp` 以及 `/proc`。
-
-使用 `mount.yaml` 定制容器文件系统。
-
-`/w` 的 `/tmp` 挂载 `tmpfs` 大小通过 `-tmp-fs-param` 指定,默认值为 `size=128m,nr_inodes=4k`
-
-如果在容器的根目录存在 `/.env` 文件,那么这个文件会在容器创建时被载入。文件的每一行会作为环境变量的初始值加入到运行程序当中。
-
-如果之后指定的挂载点目标在之前的挂载点之下,那么需要保证之前的挂载点存在目标文件或者文件夹。
-
-### 包
-
-- envexec: 核心逻辑包,在提供的环境中运行一个或多个程序
-- env: 环境的标准实现
-
-### 注意
-
-#### cgroup v2
-
-`executorserver` 目前已经支持 cgroup v2 鉴于越来越多的 Linux 发行版默认启用 cgroup v2 而不是 v1 (比如 Ubuntu 21.10+,Fedora 31+)。然而,因为 cgroup v2 在内存控制器里面缺少 `memory.max_usage_in_bytes`,内存使用量计数会转而采用 `maxrss` 指标。这项指标会显示的比使用 cgroup v1 时候要稍多,在运行使用内存较少的程序时比较明显。
-
-同时,如果本程序在容器中运行,容器中的进程会被移到 `/init` cgroup v2 控制器中来开启 cgroup v2 嵌套支持。
-
-#### CentOS 7
-
-需要开启 user 命名空间来使用 [stack overflow](https://superuser.com/questions/1294215/is-it-safe-to-enable-user-namespaces-in-centos-7-4-and-how-to-do-it/1294246#1294246)
-
-```bash
-echo user.max_user_namespaces=10000 >> /etc/sysctl.d/98-userns.conf
-sysctl -p
-# 重启生效
-reboot
-```
-
-#### 内存使用
-
-控制进程通常会使用 `20M` 内存,每个容器进程最大会使用 `20M` 内存,每个请求最大会使用 `2 * 16M` + 总 copy out max 限制 * 2 内存。请注意,缓存文件会存储在宿主机的共享内存中 (`/dev/shm`),请保证其大小足够存储运行时最大可能文件。
-
-比方说当同时请求数最大为 4 的时候,本程序最大会占用 `60 + (20+32) * 4M = 268M` + 总 copy out max 限制 * 8 内存 + 总运行程序最大内存限制。
-
-因为 go 语言 runtime 垃圾收集算法实现的问题,它并不会主动归还占用内存。这种情况可能会引发 OOM Killer 杀死进程。加入了一个后台检查线程用于在堆内存占用高时强制垃圾收集和归还内存。
-
-- `-force-gc-target` 默认 `20m`, 堆内存使用超过该值是强制垃圾收集和归还内存
-- `-force-gc-interval` 默认 `5s`, 为后台线程检查的频繁程度
-
-### 压力测试
-
-使用 `wrk` 和 `t.lua`: `wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run`.
-
-注意,这些结果只是极限情况下的表现,实际情况和使用方式相关。通常沙箱服务相比于直接运行程序,通常有 1 毫秒左右额外延迟。
-
-```lua
-wrk.method = "POST"
-wrk.body = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
-wrk.headers["Content-Type"] = "application/json;charset=UTF-8"
-```
-
-- 单线程 ~800-860 op/s Windows 10 WSL2 @ 5800X
-- 多线程 ~4500-6000 op/s Windows 10 WSL2 @ 5800X
-
-单线程:
-
-```text
-Running 30s test @ http://localhost:5050/run
- 1 threads and 1 connections
- Thread Stats Avg Stdev Max +/- Stdev
- Latency 1.16ms 132.89us 6.20ms 90.15%
- Req/Sec 0.87k 19.33 0.91k 85.33%
- Latency Distribution
- 50% 1.13ms
- 75% 1.18ms
- 90% 1.27ms
- 99% 1.61ms
- 25956 requests in 30.01s, 6.88MB read
-Requests/sec: 864.88
-Transfer/sec: 234.68KB
-```
-
### REST API 接口定义
```typescript
@@ -267,14 +87,14 @@ interface Cmd {
}
enum Status {
- Accepted = 'Accepted', // normal
- MemoryLimitExceeded = 'Memory Limit Exceeded', // mle
- TimeLimitExceeded = 'Time Limit Exceeded', // tle
- OutputLimitExceeded = 'Output Limit Exceeded', // ole
- FileError = 'File Error', // fe
- NonzeroExitStatus = 'Nonzero Exit Status',
- Signalled = 'Signalled',
- InternalError = 'Internal Error', // system error
+ Accepted = 'Accepted', // 正常情况
+ MemoryLimitExceeded = 'Memory Limit Exceeded', // 内存超限
+ TimeLimitExceeded = 'Time Limit Exceeded', // 时间超限
+ OutputLimitExceeded = 'Output Limit Exceeded', // 输出超限
+ FileError = 'File Error', // 文件错误
+ NonzeroExitStatus = 'Nonzero Exit Status', // 非 0 退出值
+ Signalled = 'Signalled', // 进程被信号终止
+ InternalError = 'Internal Error', // 内部错误
}
interface PipeIndex {
@@ -312,13 +132,13 @@ interface FileError {
}
interface Request {
- requestId?: string; // 给 WebSocket 使用
+ requestId?: string; // 给 WebSocket 使用来区分返回值的来源请求
cmd: Cmd[];
pipeMapping: PipeMap[];
}
interface CancelRequest {
- cancelRequestId: string;
+ cancelRequestId: string; // 取消某个正在进行中的请求
};
// WebSocket 请求
@@ -349,9 +169,13 @@ interface WSResult {
### 示例
-单个文件编译运行(这个例子需要安装 `g++` 例如 `apt install g++` )
+请使用 postman 或其他 REST API 调试工具向 http://localhost:5050/run 发送请求
-需要注意 `copyOutCached` 产生的文件在使用完之后需要显示删除
+单个c++文件编译运行
+
+这个例子需要安装 `g++`。如果在 docker 环境中运行,请在容器中(`docker exec -it executorserver /bin/bash`)执行 `apt update && apt install g++`
+
+需要注意,在生产环境中 `copyOutCached` 产生的文件在使用完之后需要使用(`DELETE /file/:id`)删除避免内存泄露
```json
{
@@ -376,8 +200,7 @@ interface WSResult {
}
},
"copyOut": ["stdout", "stderr"],
- "copyOutCached": ["a.cc", "a"],
- "copyOutDir": "1"
+ "copyOutCached": ["a"]
}]
}
```
@@ -395,8 +218,7 @@ interface WSResult {
"stdout": ""
},
"fileIds": {
- "a": "5LWIZAA45JHX4Y4Z",
- "a.cc": "NOHPGGDTYQUFRSLJ"
+ "a": "5LWIZAA45JHX4Y4Z"
}
}
]
@@ -422,7 +244,7 @@ interface WSResult {
"strictMemoryLimit": false,
"copyIn": {
"a": {
- "fileId": "5LWIZAA45JHX4Y4Z"
+ "fileId": "5LWIZAA45JHX4Y4Z" // 这个缓存文件的 ID 来自上一个请求返回的 fileIds
}
}
}]
@@ -558,3 +380,180 @@ interface WSResult {
```
+
+## 进阶设置
+
+### 运行要求
+
+- Linux 内核版本 >= 3.10
+- 系统 Cgroup 文件系统挂载于 `/sys/fs/cgroup`(Systemd 默认)
+
+### 系统架构
+
+```text
++----------------------------------------------------------------------------+
+| 传输层 (HTTP / WebSocket / FFI / ...) |
++----------------------------------------------------------------------------+
+| 工作协程 (运行环境池 和 运行环境生产者 ) |
++-----------------------------------------------------------+----------------+
+| 运行环境 | 文件存储 |
++--------------------+----------------+---------------------+----------+-----+
+| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | 共享内存 | 磁盘 |
++--------------------+----------------+---------------------+----------+-----+
+```
+
+### 命令行参数
+
+服务相关:
+
+- 默认监听地址是 `localhost:5050`,使用 `-http-addr` 指定
+- 默认 gRPC 接口处于关闭状态,使用 `-enable-grpc` 开启
+- 默认 gRPC 监听地址是 `localhost:5051` ,使用 `-grpc-addr` 指定
+- 默认日志等级是 info ,使用 `-silent` 关闭 或 使用 `-release` 开启 release 级别日志
+- 默认没有开启鉴权,使用 `-auth-token` 指定令牌鉴权
+- 默认没有开启 go 语言调试接口(`localhost:5052/debug`),使用 `-enable-debug` 开启,同时将日志层级设为 Debug
+- 默认没有开启 prometheus 监控接口,使用 `-enable-metrics` 开启 `localhost:5052/metrics`
+- 在启用 go 语言调试接口或者 prometheus 监控接口的情况下,默认监控接口为 `localhost:5052`,使用 `-monitor-addr` 指定
+
+沙箱相关:
+
+- 默认同时运行任务数为和 CPU 数量相同,使用 `-parallelism` 指定
+- 默认文件存储在内存里,使用 `-dir` 指定本地目录为文件存储
+- 默认 cgroup 的前缀为 `executor_server` ,使用 `-cgroup-prefix` 指定
+- 默认没有磁盘文件复制限制,使用 `-src-prefix` 限制 copyIn 操作文件目录前缀,使用逗号 `,` 分隔(需要绝对路径)(例如:`/bin,/usr`)
+- 默认时间和内存使用检查周期为 100 毫秒(`100ms`),使用 `-time-limit-checker-interval` 指定
+- 默认最大输出限制为 `256MiB`,使用 `-output-limit` 指定
+- 默认最大打开文件描述符为 `256`,使用 `-open-file-limit` 指定
+- 默认最大额外内存使用为 `16KiB` ,使用 `-extra-memory-limit` 指定
+- 默认最大 `copyOut` 文件大小为 `64MiB` ,使用 `-copy-out-limit` 指定
+- 使用 `-cpuset` 指定 `cpuset.cpus` (仅 Linux)
+- 默认容器用户开始区间为 10000 使用 `-container-cred-start` 指定(仅 Linux)
+ - 举例,默认情况下第 0 个容器使用 10001 作为容器用户。第 1 个容器使用 10002 作为容器用户,以此类推
+- 使用 `-enable-cpu-rate` 开启 `cpu` cgroup 来启用 `cpuRate` 控制(仅 Linux)
+ - 使用 `-cpu-cfs-period` 指定 cfs_period if cpu rate is enabled (default 100ms) (valid value: \[1ms, 1s\])
+- 使用 `-seccomp-conf` 指定 `seecomp` 过滤器(需要编译标志 `seccomp`,默认不开启)(仅 Linux)
+- 使用 `-pre-fork` 指定启动时创建的容器数量
+- 使用 `-tmp-fs-param` 指定容器内 `tmpfs` 的挂载参数(仅 Linux)
+- 使用 `-file-timeout` 指定文件存储文件最大时间。超出时间的文件将会删除。(举例 `30m`)
+- 使用 `-mount-conf` 指定沙箱文件系统挂载细节,详细请参见 `mount.yaml` (仅 Linux)
+- 使用 `-container-init-path` 指定 `cinit` 路径 (请不要使用,仅 debug) (仅 Linux)
+
+### 环境变量
+
+所有命令行参数都可以通过环境变量的形式来指定,(类似 `ES_HTTP_ADDR` 来指定 `-http-addr`)。使用 `executorserver --help` 查看所有环境变量
+
+#### 编译 docker
+
+终端中运行 `docker build -t executorserver -f Dockerfile.exec .`
+
+沙箱服务需要特权级别 docker 来创建子容器和提供 cgroup 资源限制。
+
+### 编译沙箱终端
+
+编译 `go build ./cmd/executorshell`
+
+运行 `./executorshell`,需要打开 gRPC 接口来使用。提供一个沙箱内的终端环境。
+
+### /run 接口返回状态
+
+- Accepted: 程序在资源限制内正常退出
+- Memory Limit Exceeded: 超出内存限制
+- Time Limit Exceeded:
+ - 超出 `timeLimit` 时间限制
+ - 或者超过 `clockLimit` 等待时间限制
+- Output Limit Exceeded:
+ - 超出 `pipeCollector` 限制
+ - 或者超出 `-output-limit` 最大输出限制
+- File Error:
+ - `copyIn` 指定文件不存在
+ - 或者 `copyIn` 指定文件大小超出沙箱文件系统限制
+ - 或者 `copyOut` 指定文件不存在
+- Non Zero Exit Status: 程序用非 0 返回值退出
+- Signalled: 程序收到结束信号而退出(例如 `SIGSEGV`)
+- Dangerous Syscall: 程序被 `seccomp` 过滤器结束
+- Internal Error:
+ - 指定程序路径不存在
+ - 或者容器创建失败
+ - 比如使用非特权 docker
+ - 或者在个人目录下以 root 权限运行
+ - 或者其他错误
+
+### 容器的文件系统
+
+在 Linux 平台,默认只读挂载点包括主机的 `/lib`, `/lib64`, `/usr`, `/bin`, `/etc/ld.so.cache`, `/etc/alternatives`, `/etc/fpc.cfg`, `/dev/null`, `/dev/urandom`, `/dev/random`, `/dev/zero`, `/dev/full` 和临时文件系统 `/w`, `/tmp` 以及 `/proc`。
+
+使用 `mount.yaml` 定制容器文件系统。
+
+`/w` 的 `/tmp` 挂载 `tmpfs` 大小通过 `-tmp-fs-param` 指定,默认值为 `size=128m,nr_inodes=4k`
+
+如果在容器的根目录存在 `/.env` 文件,那么这个文件会在容器创建时被载入。文件的每一行会作为环境变量的初始值加入到运行程序当中。
+
+如果之后指定的挂载点目标在之前的挂载点之下,那么需要保证之前的挂载点存在目标文件或者文件夹。
+
+### 包
+
+- envexec: 核心逻辑包,在提供的环境中运行一个或多个程序
+- env: 环境的标准实现
+
+### 注意
+
+#### cgroup v2
+
+`executorserver` 目前已经支持 cgroup v2 鉴于越来越多的 Linux 发行版默认启用 cgroup v2 而不是 v1 (比如 Ubuntu 21.10+,Fedora 31+)。然而,因为 cgroup v2 在内存控制器里面缺少 `memory.max_usage_in_bytes`,内存使用量计数会转而采用 `maxrss` 指标。这项指标会显示的比使用 cgroup v1 时候要稍多,在运行使用内存较少的程序时比较明显。
+
+同时,如果本程序在容器中运行,容器中的进程会被移到 `/init` cgroup v2 控制器中来开启 cgroup v2 嵌套支持。
+
+#### CentOS 7
+
+需要开启 user 命名空间来使用 [stack overflow](https://superuser.com/questions/1294215/is-it-safe-to-enable-user-namespaces-in-centos-7-4-and-how-to-do-it/1294246#1294246)
+
+```bash
+echo user.max_user_namespaces=10000 >> /etc/sysctl.d/98-userns.conf
+sysctl -p
+# 重启生效
+reboot
+```
+
+#### 内存使用
+
+控制进程通常会使用 `20M` 内存,每个容器进程最大会使用 `20M` 内存,每个请求最大会使用 `2 * 16M` + 总 copy out max 限制 * 2 内存。请注意,缓存文件会存储在宿主机的共享内存中 (`/dev/shm`),请保证其大小足够存储运行时最大可能文件。
+
+比方说当同时请求数最大为 4 的时候,本程序最大会占用 `60 + (20+32) * 4M = 268M` + 总 copy out max 限制 * 8 内存 + 总运行程序最大内存限制。
+
+因为 go 语言 runtime 垃圾收集算法实现的问题,它并不会主动归还占用内存。这种情况可能会引发 OOM Killer 杀死进程。加入了一个后台检查线程用于在堆内存占用高时强制垃圾收集和归还内存。
+
+- `-force-gc-target` 默认 `20m`, 堆内存使用超过该值是强制垃圾收集和归还内存
+- `-force-gc-interval` 默认 `5s`, 为后台线程检查的频繁程度
+
+### 压力测试
+
+使用 `wrk` 和 `t.lua`: `wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run`.
+
+注意,这些结果只是极限情况下的表现,实际情况和使用方式相关。通常沙箱服务相比于直接运行程序,通常有 1 毫秒左右额外延迟。
+
+```lua
+wrk.method = "POST"
+wrk.body = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
+wrk.headers["Content-Type"] = "application/json;charset=UTF-8"
+```
+
+- 单线程 ~800-860 op/s Windows 10 WSL2 @ 5800X
+- 多线程 ~4500-6000 op/s Windows 10 WSL2 @ 5800X
+
+单线程:
+
+```text
+Running 30s test @ http://localhost:5050/run
+ 1 threads and 1 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 1.16ms 132.89us 6.20ms 90.15%
+ Req/Sec 0.87k 19.33 0.91k 85.33%
+ Latency Distribution
+ 50% 1.13ms
+ 75% 1.18ms
+ 90% 1.27ms
+ 99% 1.61ms
+ 25956 requests in 30.01s, 6.88MB read
+Requests/sec: 864.88
+Transfer/sec: 234.68KB
+```
diff --git a/README.md b/README.md
index c1f0a89..10d220c 100644
--- a/README.md
+++ b/README.md
@@ -4,34 +4,25 @@
[中文文档](README.cn.md)
-## Executor Service
-
Fast, Simple, Secure
-### Prerequisite
+## Quick Start
-- Linux Kernel Version >= 3.10
-- Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd
+### Install & Run
-### Architecture
+Download compiled executable `executorserver` for your platform from [Release](https://github.com/criyle/go-judge/releases) and run.
-```text
-+----------------------------------------------------------------------------------+
-| Transport Layer (HTTP / WebSocket / FFI / ...) |
-+----------------------------------------------------------------------------------+
-| Executor Worker (Environment Pool w/ Environment Builder ) |
-+-----------------------------------------------------------+----------------------+
-| EnvExec | File Store |
-+--------------------+----------------+---------------------+---------------+------+
-| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Shared Memory | Disk |
-+--------------------+----------------+---------------------+---------------+------+
+Or, by docker
+
+```bash
+docker run -it --rm --privileged --shm-size=256m -p 5050:5050 --name=executorserver criyle/executorserver
```
### REST API
-A REST service to run program in restricted environment and it is basically a wrapper for `envexec` to run single / multiple programs.
+A REST service to run program in restricted environment (Listening on `localhost:5050` by default).
-- /run POST execute program in the restricted environment (examples below)
+- **/run POST execute program in the restricted environment (examples below)**
- /file GET list all cached file id to original name map
- /file POST prepare a file in the executor service (in memory), returns fileId (can be referenced in /run parameter)
- /file/:fileId GET downloads file from executor service (in memory), returns file content
@@ -40,218 +31,6 @@ A REST service to run program in restricted environment and it is basically a wr
- /version gets build git version (e.g. `v1.4.0`) together with runtime information (go version, os, platform)
- /config gets some configuration (e.g. `fileStorePath`, `runnerConfig`) together with some supported features
-Monitor HTTP endpoint (default `:5052`, specified by `-monitor-addr`)
-
-- /metrics prometheus metrics (specifies `ES_ENABLE_METRICS=1` environment variable to enable metrics)
-- /debug (specifies `ES_ENABLE_DEBUG=1` environment variable to enable go runtime debug endpoint)
-
-### Command Line Arguments
-
-Server:
-
-- The default binding address for the executor server is `:5050`. Can be specified with `-http-addr` flag.
-- By default gRPC endpoint is disabled, to enable gRPC endpoint, add `-enable-grpc` flag.
-- The default binding address for the gRPC executor server is `:5051`. Can be specified with `-grpc-addr` flag.
-- The default log level is info, use `-silent` to disable logs or use `-release` to enable release logger (auto turn on if in docker).
-- `-auth-token` to add token-based authentication to REST / gRPC
-- By default, the GO debug endpoints are disabled, to enable, specifies `-enable-debug`, and it also enables debug log
-- By default, the prometheus metrics endpoints are disabled, to enable, specifies `-enable-metrics`
-- Monitoring HTTP endpoint is enabled if metrics / debug is enabled, the default addr is `:5052` and can be specified by `-monitor-addr`
-
-Sandbox:
-
-- The default concurrency equal to number of CPU, Can be specified with `-parallelism` flag.
-- The default file store is in memory, local cache can be specified with `-dir` flag.
-- The default CGroup prefix is `executor_server`, Can be specified with `-cgroup-prefix` flag.
-- `-src-prefix` to restrict `src` copyIn path split by comma (need to be absolute path) (example: `/bin,/usr`)
-- `-time-limit-checker-interval` specifies time limit checker interval (default 100ms) (valid value: \[1ms, 1s\])
-- `-output-limit` specifies size limit of POSIX rlimit of output (default 256MiB)
-- `-extra-memory-limit` specifies the additional memory limit to check memory limit exceeded (default 16KiB)
-- `-copy-out-limit` specifies the default file copy out max (default 64MiB)
-- `-open-file-limit` specifies the max number of open files (default 256)
-- `-cpuset` specifies `cpuset.cpus` cgroup for each container (Linux only)
-- `-container-cred-start` specifies container `setuid` / `setgid` credential start point (default: 10000) (Linux only)
- - for example, by default container 0 will run with 10001 uid & gid and container 1 will run with 10002 uid & gid...
-- `-enable-cpu-rate` enabled `cpu` cgroup to control cpu rate using cfs_quota & cfs_period control (Linux only)
- - `-cpu-cfs-period` specifies cfs_period if cpu rate is enabled (default 100ms) (valid value: \[1ms, 1s\])
-- `-seccomp-conf` specifies `seecomp` filter setting to load when running program (need build tag `seccomp`) (Linux only)
- - for example, by `strace -c prog` to get all `syscall` needed and restrict to that sub set
- - however, the `syscall` count in one platform(e.g. x86_64) is not suitable for all platform, so this option is not recommended
- - the program killed by seccomp filter will have status `Dangerous Syscall`
-- `-pre-fork` specifies number of container to create when server starts
-- `-tmp-fs-param` specifies the tmpfs parameter for `/w` and `/tmp` when using default mounting (Linux only)
-- `-file-timeout` specifies maximum TTL for file created in file store (e.g. `30m`)
-- `-mount-conf` specifies detailed mount configuration, please refer `mount.yaml` as a reference (Linux only)
-- `-container-init-path` specifies path to `cinit` (do not use, debug only) (Linux only)
-
-### Environment Variables
-
-Environment variable will be override by command line arguments if they both present and all command line arguments have its correspond environment variable (e.g. `ES_HTTP_ADDR`). Run `executorserver --help` to see all the environment variable configurations.
-
-### Install & Run
-
-Download compiled executable from [Release](https://github.com/criyle/go-judge/releases) and run.
-
-Or, by docker
-
-```bash
-docker run -it --rm --privileged --shm-size=256m -p 5050:5050 criyle/executorserver
-```
-
-#### Build Executor Server
-
-Build by your own `docker build -t executorserver -f Dockerfile.exec .`
-
-The `executorserver` need root privilege to create `cgroup`. Either creates sub-directory `/sys/fs/cgroup/cpuacct/executor_server`, `/sys/fs/cgroup/memory/executor_server`, `/sys/fs/cgroup/pids/executor_server` and make execution user readable or use `sudo` to run it.
-
-#### Build Shared object
-
-Build container init `cinit`:
-
-`go build -o cinit ./cmd/cinit`
-
-Build `executor_server.so`:
-
-`go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/`
-
-For example, in JavaScript, run with `ffi-napi` (seems node 14 is not supported yet):
-
-### Build Executor Proxy
-
-Build `go build ./cmd/executorproxy`
-
-Run `./executorproxy`, connect to gRPC endpoint expose as a REST endpoint.
-
-### Build Executor Shell
-
-Build `go build ./cmd/executorshell`
-
-Run `./executorshell`, connect to gRPC endpoint with interactive shell.
-
-### Return Status
-
-- Accepted: Program exited with status code 0 within time & memory limits
-- Memory Limit Exceeded: Program uses more memory than memory limits
-- Time Limit Exceeded:
- - Program uses more CPU time than cpuLimit
- - Or, program uses more clock time than clockLimit
-- Output Limit Exceeded:
- - Program output more than pipeCollector limits
- - Or, program output more than output-limit
-- File Error:
- - CopyIn file is not existed
- - Or, CopyIn file too large for container file system
- - Or, CopyOut file is not existed after program exited
-- Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
-- Signalled: Program exited with signal (e.g. SIGSEGV)
-- Dangerous Syscall: Program killed by seccomp filter
-- Internal Error:
- - Program is not exist
- - Or, container create not successful (e.g. not privileged docker)
- - Or, other errors
-
-### Container Root Filesystem
-
-For linux platform, the default mounts points are bind mounting host's `/lib`, `/lib64`, `/usr`, `/bin`, `/etc/ld.so.cache`, `/etc/alternatives`, `/etc/fpc.cfg`, `/dev/null`, `/dev/urandom`, `/dev/random`, `/dev/zero`, `/dev/full` and mounts tmpfs at `/w`, `/tmp` and creates `/proc`.
-
-To customize mount points, please look at example `mount.yaml` file.
-
-`tmpfs` size for `/w` and `/tmp` is configured through `-tmp-fs-param` with default value `size=128m,nr_inodes=4k`
-
-If a file named `/.env` exists in the container rootfs, the container will load the file as environment variable line by line.
-
-If a bind mount is specifying a target within the previous mounted one, please ensure the target exists in the previous mount point.
-
-### Packages
-
-- envexec: run single / group of programs in parallel within restricted environment and resource constraints
-- env: reference implementation environments to inject into envexec
-
-### Windows Support
-
-- Build `executorserver` by: `go build ./cmd/executorserver/`
-- Build `executor_server.dll`: (need to install `gcc` as well) `go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/`
-- Run: `./executorserver`
-
-#### Windows Security
-
-- Resources are limited by [JobObject](https://docs.microsoft.com/en-us/windows/win32/procthread/job-objects)
-- Privillege are limited by [Restricted Low Mandatory Level Token](https://docs.microsoft.com/en-us/windows/win32/secauthz/access-tokens)
-- Low Mandatory Level directory is created for read / write
-
-### MacOS Support
-
-- Build `executorserver` by: `go build ./cmd/executorserver/`
-- Build `executor_server.dylib`: (need to install `XCode`) `go build -buildmode=c-shared -o executor_server.dylib ./cmd/ffi/`
-- Run: `./executorserver`
-
-#### MacOS Security
-
-- `sandbox-init` profile deny network access and file read / write and read / write to `/Users` directory
-
-### Notice
-
-#### cgroup v2 support
-
-The cgroup v2 is supported by `executorserver` now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, due to missing `memory.max_usage_in_bytes` in `memory` controller, the memory usage is now accounted by `maxrss` returned by `wait4` syscall. Thus, the memory usage appears higher than those who uses cgroup v1.
-
-When running in containers, the `executorserver` will migrate all processed into `/init` hierarchy to enable nesting support.
-
-#### CentOS 7
-
-By default, user namespace is disabled and it can be enabled following [stack overflow](https://superuser.com/questions/1294215/is-it-safe-to-enable-user-namespaces-in-centos-7-4-and-how-to-do-it/1294246#1294246)
-
-```bash
-echo user.max_user_namespaces=10000 >> /etc/sysctl.d/98-userns.conf
-sysctl -p
-# reboot make the config effective
-reboot
-```
-
-#### Memory Usage
-
-The controller will consume `20M` memory and each container will consume `20M` + size of tmpfs `2 * 128M`. For each request, it consumes as much as user program limit + extra limit (`16k`) + total copy out max. Notice that the cached file stores in the shared memory (`/dev/shm`) of the host, so please ensure enough size allocated.
-
-For example, when concurrency = 4, the executor itself can consume as much as `60 + (20+32) * 4M = 268M` + 4 * total copy out + total max memory of requests.
-
-Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. The background worker was introduced to checks heap usage and invokes GC when necessary.
-
-- `-force-gc-target` default `20m`, the minimal size to trigger GC
-- `-force-gc-interval` default `5s`, the interval to check memory usage
-
-### Benchmark
-
-By `wrk` with `t.lua`: `wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run`.
-
-However, these results are not the real use cases since the running time depends on the actual program specifies in the request. Normally, the executor server consumes ~1ms more compare to running without sandbox.
-
-```lua
-wrk.method = "POST"
-wrk.body = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
-wrk.headers["Content-Type"] = "application/json;charset=UTF-8"
-```
-
-- Single thread ~800-860 op/s Windows 10 WSL2 @ 5800X
-- Multi thread ~4500-6000 op/s Windows 10 WSL2 @ 5800X
-
-Single thread:
-
-```text
-Running 30s test @ http://localhost:5050/run
- 1 threads and 1 connections
- Thread Stats Avg Stdev Max +/- Stdev
- Latency 1.16ms 132.89us 6.20ms 90.15%
- Req/Sec 0.87k 19.33 0.91k 85.33%
- Latency Distribution
- 50% 1.13ms
- 75% 1.18ms
- 90% 1.27ms
- 99% 1.61ms
- 25956 requests in 30.01s, 6.88MB read
-Requests/sec: 864.88
-Transfer/sec: 234.68KB
-```
-
### REST API Interface
```typescript
@@ -525,6 +304,8 @@ Output:
+Plese use PostMan or similar tools to send request to `http://localhost:5050/run`
+
Single (this example require `apt install g++` inside the container)
```json
@@ -781,3 +562,222 @@ Output:
```
+
+## Documentation
+
+### Prerequisite
+
+- Linux Kernel Version >= 3.10
+- Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd
+
+### Architecture
+
+```text
++----------------------------------------------------------------------------------+
+| Transport Layer (HTTP / WebSocket / FFI / ...) |
++----------------------------------------------------------------------------------+
+| Executor Worker (Environment Pool w/ Environment Builder ) |
++-----------------------------------------------------------+----------------------+
+| EnvExec | File Store |
++--------------------+----------------+---------------------+---------------+------+
+| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Shared Memory | Disk |
++--------------------+----------------+---------------------+---------------+------+
+```
+
+### Command Line Arguments
+
+Server:
+
+- The default binding address for the executor server is `localhost:5050`. Can be specified with `-http-addr` flag.
+- By default gRPC endpoint is disabled, to enable gRPC endpoint, add `-enable-grpc` flag.
+- The default binding address for the gRPC executor server is `localhost:5051`. Can be specified with `-grpc-addr` flag.
+- The default log level is info, use `-silent` to disable logs or use `-release` to enable release logger (auto turn on if in docker).
+- `-auth-token` to add token-based authentication to REST / gRPC
+- By default, the GO debug endpoints (`localhost:5052/debug`) are disabled, to enable, specifies `-enable-debug`, and it also enables debug log
+- By default, the prometheus metrics endpoints (`localhost:5052/metrics`) are disabled, to enable, specifies `-enable-metrics`
+- Monitoring HTTP endpoint is enabled if metrics / debug is enabled, the default addr is `localhost:5052` and can be specified by `-monitor-addr`
+
+Sandbox:
+
+- The default concurrency equal to number of CPU, Can be specified with `-parallelism` flag.
+- The default file store is in memory, local cache can be specified with `-dir` flag.
+- The default CGroup prefix is `executor_server`, Can be specified with `-cgroup-prefix` flag.
+- `-src-prefix` to restrict `src` copyIn path split by comma (need to be absolute path) (example: `/bin,/usr`)
+- `-time-limit-checker-interval` specifies time limit checker interval (default 100ms) (valid value: \[1ms, 1s\])
+- `-output-limit` specifies size limit of POSIX rlimit of output (default 256MiB)
+- `-extra-memory-limit` specifies the additional memory limit to check memory limit exceeded (default 16KiB)
+- `-copy-out-limit` specifies the default file copy out max (default 64MiB)
+- `-open-file-limit` specifies the max number of open files (default 256)
+- `-cpuset` specifies `cpuset.cpus` cgroup for each container (Linux only)
+- `-container-cred-start` specifies container `setuid` / `setgid` credential start point (default: 10000) (Linux only)
+ - for example, by default container 0 will run with 10001 uid & gid and container 1 will run with 10002 uid & gid...
+- `-enable-cpu-rate` enabled `cpu` cgroup to control cpu rate using cfs_quota & cfs_period control (Linux only)
+ - `-cpu-cfs-period` specifies cfs_period if cpu rate is enabled (default 100ms) (valid value: \[1ms, 1s\])
+- `-seccomp-conf` specifies `seecomp` filter setting to load when running program (need build tag `seccomp`) (Linux only)
+ - for example, by `strace -c prog` to get all `syscall` needed and restrict to that sub set
+ - however, the `syscall` count in one platform(e.g. x86_64) is not suitable for all platform, so this option is not recommended
+ - the program killed by seccomp filter will have status `Dangerous Syscall`
+- `-pre-fork` specifies number of container to create when server starts
+- `-tmp-fs-param` specifies the tmpfs parameter for `/w` and `/tmp` when using default mounting (Linux only)
+- `-file-timeout` specifies maximum TTL for file created in file store (e.g. `30m`)
+- `-mount-conf` specifies detailed mount configuration, please refer `mount.yaml` as a reference (Linux only)
+- `-container-init-path` specifies path to `cinit` (do not use, debug only) (Linux only)
+
+### Environment Variables
+
+Environment variable will be override by command line arguments if they both present and all command line arguments have its correspond environment variable (e.g. `ES_HTTP_ADDR`). Run `executorserver --help` to see all the environment variable configurations.
+
+#### Build Executor Server
+
+Build by your own `docker build -t executorserver -f Dockerfile.exec .`
+
+The `executorserver` need root privilege to create `cgroup`. Either creates sub-directory `/sys/fs/cgroup/cpuacct/executor_server`, `/sys/fs/cgroup/memory/executor_server`, `/sys/fs/cgroup/pids/executor_server` and make execution user readable or use `sudo` to run it.
+
+#### Build Shared object
+
+Build container init `cinit`:
+
+`go build -o cinit ./cmd/cinit`
+
+Build `executor_server.so`:
+
+`go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/`
+
+For example, in JavaScript, run with `ffi-napi` (seems node 14 is not supported yet):
+
+### Build Executor Proxy
+
+Build `go build ./cmd/executorproxy`
+
+Run `./executorproxy`, connect to gRPC endpoint expose as a REST endpoint.
+
+### Build Executor Shell
+
+Build `go build ./cmd/executorshell`
+
+Run `./executorshell`, connect to gRPC endpoint with interactive shell.
+
+### Return Status
+
+- Accepted: Program exited with status code 0 within time & memory limits
+- Memory Limit Exceeded: Program uses more memory than memory limits
+- Time Limit Exceeded:
+ - Program uses more CPU time than cpuLimit
+ - Or, program uses more clock time than clockLimit
+- Output Limit Exceeded:
+ - Program output more than pipeCollector limits
+ - Or, program output more than output-limit
+- File Error:
+ - CopyIn file is not existed
+ - Or, CopyIn file too large for container file system
+ - Or, CopyOut file is not existed after program exited
+- Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
+- Signalled: Program exited with signal (e.g. SIGSEGV)
+- Dangerous Syscall: Program killed by seccomp filter
+- Internal Error:
+ - Program is not exist
+ - Or, container create not successful (e.g. not privileged docker)
+ - Or, other errors
+
+### Container Root Filesystem
+
+For linux platform, the default mounts points are bind mounting host's `/lib`, `/lib64`, `/usr`, `/bin`, `/etc/ld.so.cache`, `/etc/alternatives`, `/etc/fpc.cfg`, `/dev/null`, `/dev/urandom`, `/dev/random`, `/dev/zero`, `/dev/full` and mounts tmpfs at `/w`, `/tmp` and creates `/proc`.
+
+To customize mount points, please look at example `mount.yaml` file.
+
+`tmpfs` size for `/w` and `/tmp` is configured through `-tmp-fs-param` with default value `size=128m,nr_inodes=4k`
+
+If a file named `/.env` exists in the container rootfs, the container will load the file as environment variable line by line.
+
+If a bind mount is specifying a target within the previous mounted one, please ensure the target exists in the previous mount point.
+
+### Packages
+
+- envexec: run single / group of programs in parallel within restricted environment and resource constraints
+- env: reference implementation environments to inject into envexec
+
+### Windows Support
+
+- Build `executorserver` by: `go build ./cmd/executorserver/`
+- Build `executor_server.dll`: (need to install `gcc` as well) `go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/`
+- Run: `./executorserver`
+
+#### Windows Security
+
+- Resources are limited by [JobObject](https://docs.microsoft.com/en-us/windows/win32/procthread/job-objects)
+- Privillege are limited by [Restricted Low Mandatory Level Token](https://docs.microsoft.com/en-us/windows/win32/secauthz/access-tokens)
+- Low Mandatory Level directory is created for read / write
+
+### MacOS Support
+
+- Build `executorserver` by: `go build ./cmd/executorserver/`
+- Build `executor_server.dylib`: (need to install `XCode`) `go build -buildmode=c-shared -o executor_server.dylib ./cmd/ffi/`
+- Run: `./executorserver`
+
+#### MacOS Security
+
+- `sandbox-init` profile deny network access and file read / write and read / write to `/Users` directory
+
+### Notice
+
+#### cgroup v2 support
+
+The cgroup v2 is supported by `executorserver` now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, due to missing `memory.max_usage_in_bytes` in `memory` controller, the memory usage is now accounted by `maxrss` returned by `wait4` syscall. Thus, the memory usage appears higher than those who uses cgroup v1.
+
+When running in containers, the `executorserver` will migrate all processed into `/init` hierarchy to enable nesting support.
+
+#### CentOS 7
+
+By default, user namespace is disabled and it can be enabled following [stack overflow](https://superuser.com/questions/1294215/is-it-safe-to-enable-user-namespaces-in-centos-7-4-and-how-to-do-it/1294246#1294246)
+
+```bash
+echo user.max_user_namespaces=10000 >> /etc/sysctl.d/98-userns.conf
+sysctl -p
+# reboot make the config effective
+reboot
+```
+
+#### Memory Usage
+
+The controller will consume `20M` memory and each container will consume `20M` + size of tmpfs `2 * 128M`. For each request, it consumes as much as user program limit + extra limit (`16k`) + total copy out max. Notice that the cached file stores in the shared memory (`/dev/shm`) of the host, so please ensure enough size allocated.
+
+For example, when concurrency = 4, the executor itself can consume as much as `60 + (20+32) * 4M = 268M` + 4 * total copy out + total max memory of requests.
+
+Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. The background worker was introduced to checks heap usage and invokes GC when necessary.
+
+- `-force-gc-target` default `20m`, the minimal size to trigger GC
+- `-force-gc-interval` default `5s`, the interval to check memory usage
+
+### Benchmark
+
+By `wrk` with `t.lua`: `wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run`.
+
+However, these results are not the real use cases since the running time depends on the actual program specifies in the request. Normally, the executor server consumes ~1ms more compare to running without sandbox.
+
+```lua
+wrk.method = "POST"
+wrk.body = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
+wrk.headers["Content-Type"] = "application/json;charset=UTF-8"
+```
+
+- Single thread ~800-860 op/s Windows 10 WSL2 @ 5800X
+- Multi thread ~4500-6000 op/s Windows 10 WSL2 @ 5800X
+
+Single thread:
+
+```text
+Running 30s test @ http://localhost:5050/run
+ 1 threads and 1 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 1.16ms 132.89us 6.20ms 90.15%
+ Req/Sec 0.87k 19.33 0.91k 85.33%
+ Latency Distribution
+ 50% 1.13ms
+ 75% 1.18ms
+ 90% 1.27ms
+ 99% 1.61ms
+ 25956 requests in 30.01s, 6.88MB read
+Requests/sec: 864.88
+Transfer/sec: 234.68KB
+```
+
diff --git a/go.mod b/go.mod
index 0f9014d..a9ce6c3 100644
--- a/go.mod
+++ b/go.mod
@@ -14,25 +14,26 @@ require (
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/koding/multiconfig v0.0.0-20171124222453-69c27309b2d7
- github.com/prometheus/client_golang v1.15.1
+ github.com/prometheus/client_golang v1.16.0
github.com/zsais/go-gin-prometheus v0.1.0
go.uber.org/zap v1.24.0
- golang.org/x/crypto v0.9.0
- golang.org/x/net v0.10.0
- golang.org/x/sync v0.2.0
- golang.org/x/sys v0.8.0
- google.golang.org/grpc v1.55.0
- google.golang.org/protobuf v1.30.0
+ golang.org/x/crypto v0.11.0
+ golang.org/x/net v0.12.0
+ golang.org/x/sync v0.3.0
+ golang.org/x/sys v0.10.0
+ google.golang.org/grpc v1.56.2
+ google.golang.org/protobuf v1.31.0
gopkg.in/yaml.v2 v2.4.0
)
require (
- cloud.google.com/go/compute v1.19.0 // indirect
+ cloud.google.com/go/compute v1.22.0 // indirect
github.com/BurntSushi/toml v1.3.2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
- github.com/bytedance/sonic v1.9.1 // indirect
+ github.com/bytedance/sonic v1.10.0-rc // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
- github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
+ github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d // indirect
+ github.com/chenzhuoyu/iasm v0.9.0 // indirect
github.com/fatih/camelcase v1.0.0 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
@@ -48,10 +49,10 @@ require (
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
- github.com/pelletier/go-toml/v2 v2.0.8 // indirect
+ github.com/pelletier/go-toml/v2 v2.0.9 // indirect
github.com/prometheus/client_model v0.4.0 // indirect
github.com/prometheus/common v0.44.0 // indirect
- github.com/prometheus/procfs v0.10.1 // indirect
+ github.com/prometheus/procfs v0.11.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
@@ -59,10 +60,10 @@ require (
go.opentelemetry.io/otel/trace v1.16.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
- golang.org/x/arch v0.3.0 // indirect
- golang.org/x/term v0.8.0 // indirect
- golang.org/x/text v0.9.0 // indirect
- google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
+ golang.org/x/arch v0.4.0 // indirect
+ golang.org/x/term v0.10.0 // indirect
+ golang.org/x/text v0.11.0 // indirect
+ google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
diff --git a/go.sum b/go.sum
index d0dcc48..620c556 100644
--- a/go.sum
+++ b/go.sum
@@ -1,7 +1,7 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
-cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys=
-cloud.google.com/go/compute v1.19.0 h1:+9zda3WGgW1ZSTlVppLCYFIr48Pa35q1uG2N1itbCEQ=
-cloud.google.com/go/compute v1.19.0/go.mod h1:rikpw2y+UMidAe9tISo04EHNOIf42RLYF/q8Bs93scU=
+cloud.google.com/go v0.110.2 h1:sdFPBr6xG9/wkBbfhmUz/JmZC7X6LavQgcrVINrKiVA=
+cloud.google.com/go/compute v1.22.0 h1:cB8R6FtUtT1TYGl5R3xuxnW6OUIc/DrT2aiR16TTG7Y=
+cloud.google.com/go/compute v1.22.0/go.mod h1:4tCnrn48xsqlwSAiLf1HXMQk8CONslYbdiEZc9FEIbM=
cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8=
@@ -11,14 +11,17 @@ github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZx
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
-github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
-github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
+github.com/bytedance/sonic v1.10.0-rc h1:3S5HeWxjX08CUqNrXtEittExpJsEKBNzrV5UnrzHxVQ=
+github.com/bytedance/sonic v1.10.0-rc/go.mod h1:ElCzW+ufi8qKqNW0FY314xriJhyJhuoJ3gFZdAHF7NM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
-github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
+github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d h1:77cEq6EriyTZ0g/qfRdp61a3Uu/AWrgIq2s0ClJV1g0=
+github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d/go.mod h1:8EPpVsBuRksnlj1mLy4AWzRNQYxauNi62uWcE3to6eA=
+github.com/chenzhuoyu/iasm v0.9.0 h1:9fhXjVzq5hUy2gkhhgHl95zG2cEAhw9OSGs8toWWAwo=
+github.com/chenzhuoyu/iasm v0.9.0/go.mod h1:Xjy2NpN3h7aUqeqM+woSuuvxmIe6+DDsiNLIrkAmYog=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
@@ -98,6 +101,7 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg=
github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
+github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
github.com/koding/multiconfig v0.0.0-20171124222453-69c27309b2d7 h1:SWlt7BoQNASbhTUD0Oy5yysI2seJ7vWuGUp///OM4TM=
github.com/koding/multiconfig v0.0.0-20171124222453-69c27309b2d7/go.mod h1:Y2SaZf2Rzd0pXkLVhLlCiAXFCLSXAIbTKDivVgff/AM=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@@ -125,22 +129,22 @@ github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjY
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pelletier/go-toml/v2 v2.0.1/go.mod h1:r9LEWfGN8R5k0VXJ+0BkIe7MYkRdwZOjgMj2KwnJFUo=
-github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
-github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
+github.com/pelletier/go-toml/v2 v2.0.9 h1:uH2qQXheeefCCkuBBSLi7jCiSmj3VRh2+Goq2N7Xxu0=
+github.com/pelletier/go-toml/v2 v2.0.9/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.15.1 h1:8tXpTmJbyH5lydzFPoxSIJ0J46jdh3tylbvM1xCv0LI=
-github.com/prometheus/client_golang v1.15.1/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk=
+github.com/prometheus/client_golang v1.16.0 h1:yk/hx9hDbrGHovbci4BY+pRMfSuuat626eFsHb7tmT8=
+github.com/prometheus/client_golang v1.16.0/go.mod h1:Zsulrv/L9oM40tJ7T815tM89lFEugiJ9HzIqaAx4LKc=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUooCfx1yqY=
github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU=
github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY=
github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY=
-github.com/prometheus/procfs v0.10.1 h1:kYK1Va/YMlutzCGazswoHKo//tZVlFpKYh+PymziUAg=
-github.com/prometheus/procfs v0.10.1/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM=
+github.com/prometheus/procfs v0.11.0 h1:5EAgkfkMl659uZPbe9AS2N68a7Cc1TJbPEuGzFuRbyk=
+github.com/prometheus/procfs v0.11.0/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
@@ -160,8 +164,8 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
-github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY=
-github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
+github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
+github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M=
@@ -193,14 +197,14 @@ go.uber.org/zap v1.23.0/go.mod h1:D+nX8jyLsMHMYrln8A0rJjFt/T/9/bGgIhAqxv5URuY=
go.uber.org/zap v1.24.0 h1:FiJd5l1UOLj0wCgbSE0rwwXHzEdAZS6hiiSnxJN/D60=
go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg=
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
-golang.org/x/arch v0.3.0 h1:02VY4/ZcO/gBOH6PUaoiptASxtXU10jazRCP865E97k=
-golang.org/x/arch v0.3.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
+golang.org/x/arch v0.4.0 h1:A8WCeEWhLwPBKNbFi5Wv5UTCBx5zzubnXDlMOFAzFMc=
+golang.org/x/arch v0.4.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g=
-golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0=
+golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA=
+golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
@@ -219,8 +223,8 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
-golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
-golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
+golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50=
+golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -230,8 +234,8 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI=
-golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E=
+golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -247,16 +251,16 @@ golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
-golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA=
+golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
-golang.org/x/term v0.8.0 h1:n5xxQn2i3PC0yLAbjTpNT85q/Kgzcr2gIoX9OrJUols=
-golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
+golang.org/x/term v0.10.0 h1:3R7pNqamzBraeqj/Tj8qt1aQ2HpmlC+Cx/qL/7hn4/c=
+golang.org/x/term v0.10.0/go.mod h1:lpqdcUyK/oCiQxvxVrppt5ggO2KCZ5QblwqPnfZ6d5o=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
-golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4=
+golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@@ -277,20 +281,20 @@ google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc h1:XSJ8Vk1SWuNr8S18z1NZSziL0CPIXLCCMDOEFtHBOFc=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771 h1:Z8qdAF9GFsmcUuWQ5KVYIpP3PCKydn/YKORnghIalu4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771/go.mod h1:TUfxEVdsvPg18p6AslUXFoLdpED4oBnGwyqk3dV1XzM=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
-google.golang.org/grpc v1.55.0 h1:3Oj82/tFSCeUrRTg/5E/7d/W5A1tj6Ky1ABAuZuv5ag=
-google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8=
+google.golang.org/grpc v1.56.2 h1:fVRFRnXvU+x6C4IlHZewvJOVHoOv1TUuQyoRsYnB4bI=
+google.golang.org/grpc v1.56.2/go.mod h1:I9bI3vqKfayGqPUAwGdOSu7kt6oIJLixfffKrpXqQ9s=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
-google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
-google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
+google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
+google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -308,4 +312,5 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=