| .github/workflows | ||
| cmd | ||
| env | ||
| file | ||
| filestore | ||
| pb | ||
| pkg | ||
| worker | ||
| .air.conf | ||
| .air.toml | ||
| .dockerignore | ||
| .gitignore | ||
| containerPasswd.txt | ||
| Dockerfile.alpine | ||
| Dockerfile.exec | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| mount.yaml | ||
| README.md | ||
| seccomp_example.yaml | ||
go-judge
Executor Service
Architecture
Overall Architecture
+----------------------------------------------------------------------------------+
| Transport Layer (HTTP / WebSocket / FFI / ...) |
+----------------------------------------------------------------------------------+
| Executor Worker |
+-----------------------------------------------------------+----------------------+
| EnvExec + Environment Pool + Environment Builder | File Store |
+--------------------+----------------+---------------------+--------+-------+-----+
| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Memory | Local | ... |
+--------------------+----------------+---------------------+--------+-------+-----+
A rest service to run program in restricted environment and it is basically a wrapper for pkg/envexec to run single / multiple programs.
- /run POST execute program in the restricted environment
- /file GET list all cached file
- /file POST prepare a file in the executor service (in memory), returns fileId (can be referenced in /run parameter)
- /file/:fileId GET downloads file from executor service (in memory), returns file content
- /file/:fileId DELETE delete file specified by fileId
- /ws WebSocket for /run
- /metrics prometheus metrics (specifies
METRICS=1environment variable to enable metrics) - /debug (specifies
DEBUG=1environment variable to enable go runtime debug endpoint) - /version gets build git version (e.g.
v0.6.4-1-g20d2815) together with runtime information (go version, os, platform)
Install & Run Developing Server
Install GO 1.13+ from download
go get github.com/criyle/go-judge/cmd/executorserver
~/go/bin/executorserver # or executorserver if $(GOPATH)/bin is in your $PATH
Or, by docker
docker run -it --rm --privileged -p 5050:5050 criyle/executorserver:demo
Build by your own docker build -t executorserver -f Dockerfile.exec .
The executorserver need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/executor_server, /sys/fs/cgroup/memory/executor_server, /sys/fs/cgroup/pids/executor_server and make execution user readable or use sudo to run it.
Command Line Arguments
- The default binding address for the executor server is
:5050. Can be specified with-http-addrflag. - The default binding address for the gRPC executor server is
:5051. Can be specified with-grpc-addrflag. (Notice: need to setES_ENABLE_GRPC=1environment variable to enable GRPC endpoint) - The default concurrency is
4, Can be specified with-parallelismflag. - The default file store is in memory, local cache can be specified with
-dirflag. - The default log level is debug, use
-silentto disable logs or use-releaseto enable release logger (auto turn on if in docker). - The default CGroup prefix is
executor_server, Can be specified with-cgroup-prefixflag. -auth-tokento add token-based authentication to REST / gRPC-src-prefixto restrictsrccopyIn path (need to be absolute path)-time-limit-checker-intervalspecifies time limit checker interval (default 100ms) (valid value: [1ms, 1s])-output-limitspecifies size limit of POSIX rlimit of output-cpusetspecifiescpuset.cpuscgroup for each container-container-cred-startspecifies containersetuid/setgidcredential start point (default: 10000)- for example, by default container 0 will run with 10001 uid & gid and container 1 will run with 10002 uid & gid...
-enable-cpu-rateenabledcpucgroup to control cpu rate using cfs_quota & cfs_period control-cpu-cfs-periodspecifies cfs_period if cpu rate is enabled (default 100ms) (valid value: [1ms, 1s])-seccomp-confspecifiesseecompfilter setting to load when running program- for example, by
strace -c progto get allsyscallneeded and restrict to that sub set - however, the
syscallcount in one platform(e.g. x86_64) is not suitable for all platform, so this option is not recommended - the program killed by seccomp filter will have status
Dangerous Syscall
- for example, by
Environment Variables
Environment variable will be override by command line arguments if they both present. All command line arguments have its correspond environment variable.
- The http binding address specifies as
ES_HTTP_ADDR=addr - The grpc binding address specifies as
ES_GRPC_ADDR=addr - The parallelism specifies as
ES_PARALLELISM=4 - The token specifies as
ES_AUTH_TOKEN=token ES_ENABLE_GRPC=1enables gRPCES_ENABLE_METRICS=1enables metricsES_ENABLE_DEBUG=1enables debug
Build Shared object
Build container init cinit:
go build -o cinit ./cmd/cinit
Build executor_server.so:
go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/
For example, in JavaScript, run with ffi-napi (seems node 14 is not supported yet):
Build Executor Proxy
Build go build ./cmd/executorproxy
Run ./executorproxy, connect to gRPC endpoint and offers REST endpoint.
Build Executor Shell
Build go build ./cmd/executorshell
Run ./executorshell, connect to gRPC endpoint with interactive shell.
Container Root Filesystem
- necessary lib / exec / compiler / header readonly bind mounted from current file system: /lib /lib64 /bin /usr
- work directory tmpfs mount: /w (work dir), /tmp (compiler temp files)
The following mounts point are examples that can be configured through config file later
- additional compiler scripts / exec readonly bind mounted: /c
- additional header readonly bind mounted: /i
Utilities
- pkg/envexec: run single / group of programs in parallel within restricted environment and resource constraints
- pkg/pool: reference implementation for Cgroup & Environment Pool
Windows Support
Build executorserver by:
go build ./cmd/executorserver/
Build executor_server.dll: (need to install gcc as well)
go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/
Run: ./executorserver
Windows Security
- Resources are limited by JobObject
- Privillege are limited by Restricted Low Mandatory Level Token
- Low Mandatory Level directory is created for read / write
MacOS Support
Build executorserver by:
go build ./cmd/executorserver/
Build executor_server.dylib: (need to install XCode)
go build -buildmode=c-shared -o executor_server.dylib ./cmd/ffi/
Run: ./executorserver
MacOS Security
sandbox-initprofile deny network access and file read / write
Benchmark
By wrk with t.lua:
- Tested single thread ~140-160 op/s macOS Docker Desktop & ~400-460 op/s Windows 10 WSL2.
- Tested multi thread ~1100-1200 op/s Windows 10 WSL2
wrk.method = "POST"
wrk.body = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
wrk.headers["Content-Type"] = "application/json;charset=UTF-8"
wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run
e.g.:
Running 30s test @ http://localhost:5050/run
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.17ms 446.05us 22.37ms 93.88%
Req/Sec 463.26 26.15 500.00 78.67%
Latency Distribution
50% 2.08ms
75% 2.27ms
90% 2.50ms
99% 3.13ms
13846 requests in 30.01s, 3.65MB read
Requests/sec: 461.30
Transfer/sec: 124.38KB
TODO
- Github actions to auto build
- Configure mounts using YAML config file
- Investigate root-free running mechanism (no cgroup && not set uid / gid)
- Investigate RLimit settings (cpu, data, fsize, stack, noFile)
- Add WebSocket for job submission
- Windows support
- MacOS support
- GRPC + protobuf support
- Token-based authentication
- Prometheus metrics support
- Customize container workDir, hostName & domainName
API interface
interface LocalFile {
src: string; // absolute path for the file
}
interface MemoryFile {
content: string | Buffer; // file contents
}
interface PreparedFile {
fileId: string; // fileId defines file uploaded by /file
}
interface Pipe {
name: string; // file name in copyOut
max: number; // maximum bytes to collect from pipe
}
interface Cmd {
args: string[]; // command line argument
env?: string[]; // environment
// specifies file input / pipe collector for program file descriptors
files?: (LocalFile | MemoryFile | PreparedFile | Pipe | null)[];
tty?: boolean; // enables tty on the input and output pipes (should have just one input & one output)
// Notice: must have TERM environment variables (e.g. TERM=xterm)
// limitations
cpuLimit?: number; // ns
realCpuLimit?: number; // ns
memoryLimit?: number; // byte
stackLimit?: number; // byte (N/A on windows, macOS cannot set over 32M)
procLimit?: number;
// copy the correspond file to the container dst path
copyIn?: {[dst:string]:LocalFile | MemoryFile | PreparedFile};
// copy out specifies files need to be copied out from the container after execution
copyOut?: string[];
// similar to copyOut but stores file in executor service and returns fileId, later download through /file/:fileId
copyOutCached?: string[];
// specifies the directory to dump container /w content
copyOutDir: string
// specifies the max file size to copy out
copyOutMax: number; // byte
}
enum Status {
Accepted, // normal
MemoryLimitExceeded, // mle
TimeLimitExceeded, // tle
OutputLimitExceeded, // ole
FileError, // fe
RuntimeError, // re
DangerousSyscall, // dgs
InternalError, // system error
}
interface PipeIndex {
index: number; // the index of cmd
fd: number; // the fd number of cmd
}
interface PipeMap {
in: PipeIndex; // input end of the pipe
out: PipeIndex; // output end of the pipe
}
interface Request {
requestId?: string; // for WebSocket requests
cmd: Cmd[];
pipeMapping: PipeMap[];
}
interface Result {
status: Status;
error?: string; // potential system error message
time: number; // ns (cgroup recorded time)
memory: number; // byte
runTime: number; // ns (wall clock time)
// copyFile name -> content
files?: {[name:string]:string};
// copyFileCached name -> fileId
fileIds?: {[name:string]:string};
}
// WebSocket results
interface WSResult {
requestId: string;
results: []Result;
error?: string;
}
Example Request & Response
FFI:
var ffi = require('ffi-napi');
var executor_server = ffi.Library('./executor_server', {
'Init': ['int', ['string']],
'Exec': ['string', ['string']],
'FileList': ['string', []],
'FileAdd': ['string', ['string']],
'FileGet': ['string', ['string']],
'FileDelete': ['string', ['string']]
});
if (executor_server.Init(JSON.stringify({
cinitPath: "/judge/cinit",
parallelism: 4,
}))) {
console.log("Failed to init executor server");
}
const result = JSON.parse(executor_server.Exec(JSON.stringify({
"cmd": [{
"args": ["/bin/cat", "test.txt"],
"env": ["PATH=/usr/bin:/bin"],
"files": [{
"content": ""
}, {
"name": "stdout",
"max": 10240
}, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 10000000000,
"memoryLimit": 104857600,
"procLimit": 50,
"copyIn": {
"test.txt": {
"content": "TEST"
}
}
}]
})));
console.log(result);
// Async
executor_server.Exec.async(JSON.stringify({
"cmd": [{
"args": ["/bin/cat", "test.txt"],
"env": ["PATH=/usr/bin:/bin"],
"files": [{
"content": ""
}, {
"name": "stdout",
"max": 10240
}, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 10000000000,
"memoryLimit": 104857600,
"procLimit": 50,
"copyIn": {
"test.txt": {
"content": "TEST"
}
}
}]
}), (err, res) => {
if (err) throw err;
console.log(JSON.parse(res));
});
const fileAdd = (param) => new Promise((resolve, reject) => {
executor_server.FileAdd.async(JSON.stringify(param), (err, res) => {
if (err != null) { reject(err); } else { resolve(res); }
});
});
const fileList = () => new Promise((resolve, reject) => {
executor_server.FileList.async((err, res) => {
if (err != null && res == null) { reject(err); } else { resolve(JSON.parse(res)); }
});
});
const fileGet = (param) => new Promise((resolve, reject) => {
executor_server.FileGet.async(JSON.stringify(param), (err, res) => {
if (err != null && res == null) { reject(err); } else { resolve(res); }
});
});
const fileDelete = (param) => new Promise((resolve, reject) => {
executor_server.FileDelete.async(JSON.stringify(param), (err, res) => {
if (err != null && res == null) { reject(err); } else { resolve(res); }
});
});
const fileOps = async () => {
const fileId = await fileAdd({ name: 'Name', content: 'Content' });
console.log(fileId);
const list = await fileList();
console.log(list);
const file = await fileGet({ id: fileId });
console.log(file);
const d = await fileDelete({ id: fileId });
console.log(d);
const e = await fileList();
console.log(e);
};
fileOps();
Output:
{
requestId: '',
results: [
{
status: 'Accepted',
exitStatus: 0,
time: 814048,
memory: 253952,
files: [Object]
}
]
}
Single (this example require apt install g++ inside the container):
{
"cmd": [{
"args": ["/usr/bin/g++", "a.cc", "-o", "a"],
"env": ["PATH=/usr/bin:/bin"],
"files": [{
"content": ""
}, {
"name": "stdout",
"max": 10240
}, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 10000000000,
"memoryLimit": 104857600,
"procLimit": 50,
"copyIn": {
"a.cc": {
"content": "#include <iostream>\nusing namespace std;\nint main() {\nint a, b;\ncin >> a >> b;\ncout << a + b << endl;\n}"
}
},
"copyOut": ["stdout", "stderr"],
"copyOutCached": ["a.cc", "a"],
"copyOutDir": "1"
}]
}
[
{
"status": "Accepted",
"exitStatus": 0,
"time": 303225231,
"memory": 32243712,
"runTime": 524177700,
"files": {
"stderr": "",
"stdout": ""
},
"fileIds": {
"a": "5LWIZAA45JHX4Y4Z",
"a.cc": "NOHPGGDTYQUFRSLJ"
}
}
]
Multiple (interaction problem):
{
"cmd": [{
"args": ["/bin/cat", "1"],
"env": ["PATH=/usr/bin:/bin"],
"files": [{
"content": ""
}, null, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 1000000000,
"memoryLimit": 1048576,
"procLimit": 50,
"copyIn": {
"1": { "content": "TEST 1" }
},
"copyOut": ["stderr"]
},
{
"args": ["/bin/cat"],
"env": ["PATH=/usr/bin:/bin"],
"files": [null, {
"name": "stdout",
"max": 10240
}, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 1000000000,
"memoryLimit": 1048576,
"procLimit": 50,
"copyOut": ["stdout", "stderr"]
}],
"pipeMapping": [{
"in" : {"index": 0, "fd": 1 },
"out" : {"index": 1, "fd" : 0 }
}]
}
[
{
"status": "Accepted",
"exitStatus": 0,
"time": 1545123,
"memory": 253952,
"runTime": 4148800,
"files": {
"stderr": ""
},
"fileIds": {}
},
{
"status": "Accepted",
"exitStatus": 0,
"time": 1501463,
"memory": 253952,
"runTime": 5897700,
"files": {
"stderr": "",
"stdout": "TEST 1"
},
"fileIds": {}
}
]
Compile On Windows (cygwin):
{
"cmd": [{
"args": ["C:\\Cygwin\\bin\\g++", "a.cc", "-o", "a"],
"env": ["PATH=C:\\Cygwin\\bin;"],
"files": [{
"content": ""
}, {
"name": "stdout",
"max": 10240
}, {
"name": "stderr",
"max": 10240
}],
"cpuLimit": 10000000000,
"memoryLimit": 104857600,
"procLimit": 50,
"copyIn": {
"a.cc": {
"content": "#include <iostream>\n#include <signal.h>\n#include <unistd.h>\nusing namespace std;\nint main() {\nint a, b;\ncin >> a >> b;\ncout << a + b << endl;\n}"
}
},
"copyOutCached": ["a.exe"]
}]
}
[
{
"status": "Accepted",
"exitStatus": 0,
"time": 140625000,
"memory": 36286464,
"files": {
"stderr": "",
"stdout": ""
},
"fileIds": {
"a.exe": "HLQH2OF4MXUUJBCB"
}
}
]
Infinite loop with cpu rate control:
{
"cmd": [{
"args": ["/usr/bin/python3", "1.py"],
"env": ["PATH=/usr/bin:/bin"],
"files": [{"content": ""}, {"name": "stdout","max": 10240}, {"name": "stderr","max": 10240}],
"cpuLimit": 3000000000,
"realCpuLimit": 4000000000,
"memoryLimit": 104857600,
"procLimit": 50,
"cpuRate": 0.1,
"copyIn": {
"1.py": {
"content": "while True:\n pass"
}
}
}]
}
[
{
"status": "Time Limit Exceeded",
"exitStatus": 9,
"time": 414803599,
"memory": 3657728,
"runTime": 4046054900,
"files": {
"stderr": "",
"stdout": ""
}
}
]