评测机
Go to file
2020-11-29 16:13:06 -08:00
.github/workflows customizable workDir, hostName, domainName 2020-11-29 16:13:06 -08:00
cmd Refactor config parsing 2020-11-29 09:00:12 -08:00
env customizable workDir, hostName, domainName 2020-11-29 16:13:06 -08:00
file Add tty io 2020-06-16 16:44:12 -04:00
filestore Re-organize files 2020-05-18 20:14:45 -04:00
pb FIX TYPO: parallism -> parallelism 2020-09-06 00:33:16 -07:00
pkg customizable workDir, hostName, domainName 2020-11-29 16:13:06 -08:00
worker Refactor config parsing 2020-11-29 09:00:12 -08:00
.air.conf Bump dependencies versions 2020-11-28 15:21:21 -08:00
.air.toml Add Prefork & no 500 when copyin 2020-11-08 22:34:58 +00:00
.dockerignore Add docker file for alpine 2020-04-04 00:30:37 -04:00
.gitignore Bump dependencies versions 2020-11-28 15:21:21 -08:00
Dockerfile.alpine Add gRPC support 2020-05-21 18:49:31 -04:00
Dockerfile.exec fix docker file 2020-06-09 00:00:09 -04:00
go.mod Refactor config parsing 2020-11-29 09:00:12 -08:00
go.sum Refactor config parsing 2020-11-29 09:00:12 -08:00
LICENSE Initial commit 2019-08-24 15:32:38 -07:00
mount.yaml customizable workDir, hostName, domainName 2020-11-29 16:13:06 -08:00
README.md customizable workDir, hostName, domainName 2020-11-29 16:13:06 -08:00

go-judge

GoDoc Go Report Card Release Build

Executor Service

Architecture

Overall Architecture

+----------------------------------------------------------------------------------+
| Transport Layer (HTTP / WebSocket / FFI / ...)                                   |
+----------------------------------------------------------------------------------+
| Executor Worker                                                                  |
+-----------------------------------------------------------+----------------------+
| EnvExec + Environment Pool + Environment Builder          | File Store           |
+--------------------+----------------+---------------------+--------+-------+-----+
| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Memory | Local | ... |
+--------------------+----------------+---------------------+--------+-------+-----+

A rest service to run program in restricted environment and it is basically a wrapper for pkg/envexec to run single / multiple programs.

  • /run POST execute program in the restricted environment
  • /file GET list all cached file
  • /file POST prepare a file in the executor service (in memory), returns fileId (can be referenced in /run parameter)
  • /file/:fileId GET downloads file from executor service (in memory), returns file content
  • /file/:fileId DELETE delete file specified by fileId
  • /ws WebSocket for /run
  • /metrics prometheus metrics (specifies METRICS=1 environment variable to enable metrics)
  • /debug (specifies DEBUG=1 environment variable to enable go runtime debug endpoint)
  • /version gets build git version (e.g. v0.6.4-1-g20d2815) together with runtime information (go version, os, platform)

Install & Run Developing Server

Install GO 1.13+ from download

go get github.com/criyle/go-judge/cmd/executorserver
~/go/bin/executorserver # or executorserver if $(GOPATH)/bin is in your $PATH

Or, by docker

docker run -it --rm --privileged -p 5050:5050 criyle/executorserver:demo

Build by your own docker build -t executorserver -f Dockerfile.exec .

The executorserver need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/executor_server, /sys/fs/cgroup/memory/executor_server, /sys/fs/cgroup/pids/executor_server and make execution user readable or use sudo to run it.

Command Line Arguments

  • The default binding address for the executor server is :5050. Can be specified with -http-addr flag.
  • The default binding address for the gRPC executor server is :5051. Can be specified with -grpc-addr flag. (Notice: need to set ES_ENABLE_GRPC=1 environment variable to enable GRPC endpoint)
  • The default concurrency is 4, Can be specified with -parallelism flag.
  • The default file store is in memory, local cache can be specified with -dir flag.
  • The default log level is debug, use -silent to disable logs or use -release to enable release logger (auto turn on if in docker).
  • The default CGroup prefix is executor_server, Can be specified with -cgroup-prefix flag.
  • -auth-token to add token-based authentication to REST / gRPC
  • -src-prefix to restrict src copyIn path (need to be absolute path)
  • -time-limit-checker-interval specifies time limit checker interval (default 100ms)

Environment Variables

Environment variable will be override by command line arguments if they both present. All command line arguments have its correspond environment variable.

  • The http binding address specifies as ES_HTTP_ADDR=addr
  • The grpc binding address specifies as ES_GRPC_ADDR=addr
  • The parallelism specifies as ES_PARALLELISM=4
  • The token specifies as ES_AUTH_TOKEN=token
  • ES_ENABLE_GRPC=1 enables gRPC
  • ES_ENABLE_METRICS=1 enables metrics
  • ES_ENABLE_DEBUG=1 enables debug

Build Shared object

Build container init cinit:

go build -o cinit ./cmd/cinit

Build executor_server.so:

go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/

For example, in JavaScript, run with ffi-napi (seems node 14 is not supported yet):

Build Executor Proxy

Build go build ./cmd/executorproxy

Run ./executorproxy, connect to gRPC endpoint and offers REST endpoint.

Build Executor Shell

Build go build ./cmd/executorshell

Run ./executorshell, connect to gRPC endpoint with interactive shell.

Container Root Filesystem

  • necessary lib / exec / compiler / header readonly bind mounted from current file system: /lib /lib64 /bin /usr
  • work directory tmpfs mount: /w (work dir), /tmp (compiler temp files)

The following mounts point are examples that can be configured through config file later

  • additional compiler scripts / exec readonly bind mounted: /c
  • additional header readonly bind mounted: /i

Utilities

  • pkg/envexec: run single / group of programs in parallel within restricted environment and resource constraints
  • pkg/pool: reference implementation for Cgroup & Environment Pool

Windows Support

Build executorserver by:

go build ./cmd/executorserver/

Build executor_server.dll: (need to install gcc as well)

go build -buildmode=c-shared -o executor_server.so ./cmd/ffi/

Run: ./executorserver

Windows Security

MacOS Support

Build executorserver by:

go build ./cmd/executorserver/

Build executor_server.dylib: (need to install XCode)

go build -buildmode=c-shared -o executor_server.dylib ./cmd/ffi/

Run: ./executorserver

MacOS Security

  • sandbox-init profile deny network access and file read / write

Benchmark

By wrk with t.lua:

  • Tested single thread ~140-160 op/s macOS Docker Desktop & ~400-460 op/s Windows 10 WSL2.
  • Tested multi thread ~1100-1200 op/s Windows 10 WSL2
wrk.method = "POST"
wrk.body   = '{"cmd":[{"args":["/bin/cat","a.hs"],"env":["PATH=/usr/bin:/bin"],"files":[{"content":""},{"name":"stdout","max":10240},{"name":"stderr","max":10240}],"cpuLimit":10000000000,"memoryLimit":104857600,"procLimit":50,"copyIn":{"a.hs":{"content":"main = putStrLn \\"Hello, World!\\""},"b":{"content":"TEST"}}}]}'
wrk.headers["Content-Type"] = "application/json;charset=UTF-8"

wrk -s t.lua -c 1 -t 1 -d 30s --latency http://localhost:5050/run

e.g.:

Running 30s test @ http://localhost:5050/run
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.17ms  446.05us  22.37ms   93.88%
    Req/Sec   463.26     26.15   500.00     78.67%
  Latency Distribution
     50%    2.08ms
     75%    2.27ms
     90%    2.50ms
     99%    3.13ms
  13846 requests in 30.01s, 3.65MB read
Requests/sec:    461.30
Transfer/sec:    124.38KB

TODO

  • Github actions to auto build
  • Configure mounts using YAML config file
  • Investigate root-free running mechanism (no cgroup && not set uid / gid)
  • Investigate RLimit settings (cpu, data, fsize, stack, noFile)
  • Add WebSocket for job submission
  • Windows support
  • MacOS support
  • GRPC + protobuf support
  • Token-based authentication
  • Prometheus metrics support
  • Customize container workDir, hostName & domainName

API interface

interface LocalFile {
    src: string; // absolute path for the file
}

interface MemoryFile {
    content: string | Buffer; // file contents
}

interface PreparedFile {
    fileId: string; // fileId defines file uploaded by /file
}

interface Pipe {
    name: string; // file name in copyOut
    max: number;  // maximum bytes to collect from pipe
}

interface Cmd {
    args: string[]; // command line argument
    env?: string[]; // environment

    // specifies file input / pipe collector for program file descriptors
    files?: (LocalFile | MemoryFile | PreparedFile | Pipe | null)[];
    tty?: boolean; // enables tty on the input and output pipes (should have just one input & one output)
    // Notice: must have TERM environment variables (e.g. TERM=xterm)

    // limitations
    cpuLimit?: number;     // ns
    realCpuLimit?: number; // ns
    memoryLimit?: number;  // byte
    stackLimit?: number;   // byte (N/A on windows, macOS cannot set over 32M)
    procLimit?: number;

    // copy the correspond file to the container dst path
    copyIn?: {[dst:string]:LocalFile | MemoryFile | PreparedFile};

    // copy out specifies files need to be copied out from the container after execution
    copyOut?: string[];
    // similar to copyOut but stores file in executor service and returns fileId, later download through /file/:fileId
    copyOutCached?: string[];
    // specifies the directory to dump container /w content
    copyOutDir: string
    // specifies the max file size to copy out
    copyOutMax: number; // byte
}

enum Status {
    Accepted,            // normal
    MemoryLimitExceeded, // mle
    TimeLimitExceeded,   // tle
    OutputLimitExceeded, // ole
    FileError,           // fe
    RuntimeError,        // re
    DangerousSyscall,    // dgs
    InternalError,       // system error
}

interface PipeIndex {
    index: number; // the index of cmd
    fd: number;    // the fd number of cmd
}

interface PipeMap {
    in: PipeIndex;  // input end of the pipe
    out: PipeIndex; // output end of the pipe
}

interface Request {
    requestId?: string; // for WebSocket requests
    cmd: Cmd[];
    pipeMapping: PipeMap[];
}

interface Result {
    status: Status;
    error?: string; // potential system error message
    time: number;   // ns (cgroup recorded time)
    memory: number; // byte
    runTime: number; // ns (wall clock time)
    // copyFile name -> content
    files?: {[name:string]:string};
    // copyFileCached name -> fileId
    fileIds?: {[name:string]:string};
}

// WebSocket results
interface WSResult {
    requestId: string;
    results: []Result;
    error?: string;
}

Example Request & Response

FFI:

var ffi = require('ffi-napi');

var executor_server = ffi.Library('./executor_server', {
    'Init': ['int', ['string']],
    'Exec': ['string', ['string']],
    'FileList': ['string', []],
    'FileAdd': ['string', ['string']],
    'FileGet': ['string', ['string']],
    'FileDelete': ['string', ['string']]
});

if (executor_server.Init(JSON.stringify({
    cinitPath: "/judge/cinit",
    parallelism: 4,
}))) {
    console.log("Failed to init executor server");
}

const result = JSON.parse(executor_server.Exec(JSON.stringify({
    "cmd": [{
        "args": ["/bin/cat", "test.txt"],
        "env": ["PATH=/usr/bin:/bin"],
        "files": [{
            "content": ""
        }, {
            "name": "stdout",
            "max": 10240
        }, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 10000000000,
        "memoryLimit": 104857600,
        "procLimit": 50,
        "copyIn": {
            "test.txt": {
                "content": "TEST"
            }
        }
    }]
})));
console.log(result);

// Async
executor_server.Exec.async(JSON.stringify({
    "cmd": [{
        "args": ["/bin/cat", "test.txt"],
        "env": ["PATH=/usr/bin:/bin"],
        "files": [{
            "content": ""
        }, {
            "name": "stdout",
            "max": 10240
        }, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 10000000000,
        "memoryLimit": 104857600,
        "procLimit": 50,
        "copyIn": {
            "test.txt": {
                "content": "TEST"
            }
        }
    }]
}), (err, res) => {
    if (err) throw err;
    console.log(JSON.parse(res));
});

const fileAdd = (param) => new Promise((resolve, reject) => {
    executor_server.FileAdd.async(JSON.stringify(param), (err, res) => {
        if (err != null) { reject(err); } else { resolve(res); }
    });
});
const fileList = () => new Promise((resolve, reject) => {
    executor_server.FileList.async((err, res) => {
        if (err != null && res == null) { reject(err); } else { resolve(JSON.parse(res)); }
    });
});
const fileGet = (param) => new Promise((resolve, reject) => {
    executor_server.FileGet.async(JSON.stringify(param), (err, res) => {
        if (err != null && res == null) { reject(err); } else { resolve(res); }
    });
});
const fileDelete = (param) => new Promise((resolve, reject) => {
    executor_server.FileDelete.async(JSON.stringify(param), (err, res) => {
        if (err != null && res == null) { reject(err); } else { resolve(res); }
    });
});

const fileOps = async () => {
    const fileId = await fileAdd({ name: 'Name', content: 'Content' });
    console.log(fileId);
    const list = await fileList();
    console.log(list);
    const file = await fileGet({ id: fileId });
    console.log(file);
    const d = await fileDelete({ id: fileId });
    console.log(d);
    const e = await fileList();
    console.log(e);
};

fileOps();

Output:

{
  requestId: '',
  results: [
    {
      status: 'Accepted',
      exitStatus: 0,
      time: 814048,
      memory: 253952,
      files: [Object]
    }
  ]
}

Single (this example require apt install g++ inside the container):

{
    "cmd": [{
        "args": ["/usr/bin/g++", "a.cc", "-o", "a"],
        "env": ["PATH=/usr/bin:/bin"],
        "files": [{
            "content": ""
        }, {
            "name": "stdout",
            "max": 10240
        }, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 10000000000,
        "memoryLimit": 104857600,
        "procLimit": 50,
        "copyIn": {
            "a.cc": {
                "content": "#include <iostream>\nusing namespace std;\nint main() {\nint a, b;\ncin >> a >> b;\ncout << a + b << endl;\n}"
            }
        },
        "copyOut": ["stdout", "stderr"],
        "copyOutCached": ["a.cc", "a"],
        "copyOutDir": "1"
    }]
}
[
    {
        "status": "Accepted",
        "exitStatus": 0,
        "time": 303225231,
        "memory": 32243712,
        "runTime": 524177700,
        "files": {
            "stderr": "",
            "stdout": ""
        },
        "fileIds": {
            "a": "5LWIZAA45JHX4Y4Z",
            "a.cc": "NOHPGGDTYQUFRSLJ"
        }
    }
]

Multiple (interaction problem):

{
    "cmd": [{
        "args": ["/bin/cat", "1"],
        "env": ["PATH=/usr/bin:/bin"],
        "files": [{
            "content": ""
        }, null, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 1000000000,
        "memoryLimit": 1048576,
        "procLimit": 50,
        "copyIn": {
            "1": { "content": "TEST 1" }
        },
        "copyOut": ["stderr"]
    },
    {
        "args": ["/bin/cat"],
        "env": ["PATH=/usr/bin:/bin"],
        "files": [null, {
            "name": "stdout",
            "max": 10240
        }, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 1000000000,
        "memoryLimit": 1048576,
        "procLimit": 50,
        "copyOut": ["stdout", "stderr"]
    }],
    "pipeMapping": [{
        "in" : {"index": 0, "fd": 1 },
        "out" : {"index": 1, "fd" : 0 }
    }]
}
[
    {
        "status": "Accepted",
        "exitStatus": 0,
        "time": 1545123,
        "memory": 253952,
        "runTime": 4148800,
        "files": {
            "stderr": ""
        },
        "fileIds": {}
    },
    {
        "status": "Accepted",
        "exitStatus": 0,
        "time": 1501463,
        "memory": 253952,
        "runTime": 5897700,
        "files": {
            "stderr": "",
            "stdout": "TEST 1"
        },
        "fileIds": {}
    }
]

Compile On Windows (cygwin):

{
    "cmd": [{
        "args": ["C:\\Cygwin\\bin\\g++", "a.cc", "-o", "a"],
        "env": ["PATH=C:\\Cygwin\\bin;"],
        "files": [{
            "content": ""
        }, {
            "name": "stdout",
            "max": 10240
        }, {
            "name": "stderr",
            "max": 10240
        }],
        "cpuLimit": 10000000000,
        "memoryLimit": 104857600,
        "procLimit": 50,
        "copyIn": {
            "a.cc": {
                "content": "#include <iostream>\n#include <signal.h>\n#include <unistd.h>\nusing namespace std;\nint main() {\nint a, b;\ncin >> a >> b;\ncout << a + b << endl;\n}"
            }
        },
        "copyOutCached": ["a.exe"]
    }]
}
[
    {
        "status": "Accepted",
        "exitStatus": 0,
        "time": 140625000,
        "memory": 36286464,
        "files": {
            "stderr": "",
            "stdout": ""
        },
        "fileIds": {
            "a.exe": "HLQH2OF4MXUUJBCB"
        }
    }
]