go-sandbox/README.md
criyle 4f2257a187 Remove types pkg & update documents
- move types pkg into runner
- add documentations for cgroup & container
- add benchmakr for cgroup & container
2020-03-02 03:14:13 -05:00

229 lines
7.5 KiB
Markdown

# go-sandbox
[![GoDoc](https://godoc.org/github.com/criyle/go-sandbox?status.svg)](https://godoc.org/github.com/criyle/go-sandbox) [![Go Report Card](https://goreportcard.com/badge/github.com/criyle/go-sandbox)](https://goreportcard.com/report/github.com/criyle/go-sandbox) [![Release](https://img.shields.io/github/v/tag/criyle/go-sandbox)](https://github.com/criyle/go-sandbox/releases/latest)
Original goal was to replica [uoj-judger/run_program](https://github.com/vfleaking/uoj) in GO language using [libseccomp](https://github.com/pkg/seccomp/libseccomp-golang). As technology grows, it also implements new technologies including Linux namespace and cgroup.
The idea of rootfs and interval CPU usage checking comes from [syzoj/judge-v3](https://github.com/syzoj/judge-v3) and the pooled pre-forked container comes from [vijos/jd4](https://github.com/vijos/jd4).
Notice: Only works on Linux since ptrace, unshare, cgroup are available only on Linux
## Build & Install
- install latest go compiler from [golang/download](https://golang.org/dl/)
- install libseccomp library: (for Ubuntu) `apt install libseccomp-dev`
- build & install: `go install github.com/criyle/go-sandbox/...`
## Technologies
### libseccomp + ptrace (improved UOJ sandbox)
1. Restricted computing resource by POSIX rlimit: Time & Memory (Stack) & Output
2. Restricted syscall access (by libseccomp & ptrace)
3. Restricted file access (read & write & access & exec). Evaluated by UOJ FileSet
Improvements:
1. Precise resource limits (s -> ms, mb -> kb)
2. More architectures (arm32, arm64)
3. Allow multiple traced programs in different threads
4. Allow pipes as input / output files
Default file access syscall check:
- check file read / write: `open`, `openat`
- check file read: `readlink`, `readlinkat`
- check file write: `unlink`, `unlinkat`, `chmod`, `rename`
- check file access: `stat`, `lstat`, `access`, `faccessat`
- check file exec: `execve`, `execveat`
### linux namespace + cgroup
1. Unshare & bind mount rootfs based on hostfs (elimilated ptrace)
2. Use Linux Control Groups to limit & acct CPU & memory (elimilate wait4.rusage)
3. Container tech with execveat memfd, sethostname, setdomainname
## Design
### Result Status
- Normal (no error)
- Program Error
- Resource Limit Exceeded
- Time
- Memory
- Output
- Unauthorized Access
- Disallowed Syscall
- Runtime Error
- Signalled
- `SIGXCPU` / `SIGKILL` are treated as TimeLimitExceeded by rlimit or caller kill
- `SIGXFSZ` is treated as OutputLimitExceeded by rlimit
- `SIGSYS` is treaded as Disallowed Syscall by seccomp
- Potential Runtime error are: `SIGSEGV` (segment fault)
- Nonzero Exit Status
- Program Runner Error
### Result Structure
``` go
type Result struct {
Status // result status
ExitStatus int // exit status (signal number if signalled)
Error string // potential detailed error message (for program runner error)
Time time.Duration // used user CPU time (underlying type int64 in ns)
Memory Size // used user memory (underlying type uint64 in bytes)
// metrics for the program runner
SetUpTime time.Duration
RunningTime time.Duration
}
```
### Runner Interface
Configured runner to run the program. `Context` is used to cancel (control time limit exceeded event; should not be nil).
``` go
type Runner interface {
Run(context.Context) <-chan runner.Result
}
```
### Pre-forked Container Protocol
1. Pre-fork container to run programs inside
2. Unix socket to pass fd inside / outside
Container / Host Communication Protocol (single thread):
- ping (alive check):
- reply: pong
- conf (set configuration):
- reply pong
- open (open files in given mode inside container):
- send: []OpenCmd
- reply: "success", file fds / "error"
- delete (unlink file / rmdir dir inside container):
- send: path
- reply: "finished" / "error"
- reset (clean up container for later use (clear workdir / tmp)):
- send:
- reply: "success"
- execve: (execute file inside container):
- send: argv, env, rLimits, fds
- reply:
- success: "success", pid
- failed: "failed"
- send (success): "init_finished" (as cmd)
- reply: "finished" / send: "kill" (as cmd)
- send: "kill" (as cmd) / reply: "finished"
- reply:
Any socket related error will cause the container exit (with all process inside container)
### Pre-forked Container Environment
Container restricted environment is accessed though RPC interface defined by above protocol
Provides:
- File access
- Open: create / access files
- Delete: remove file
- Management
- Ping: alive check
- Reset: remove temporary files
- Destroy: destroy the container environment
- Run program
- Execve: execute program with given parameters
``` go
type Environment interface {
Ping() error
Open([]OpenCmd) ([]*os.File, error)
Delete(p string) error
Reset() error
Execve(context.Context, ExecveParam) <-chan runner.Result
Destroy() error
}
```
## Packages (/pkg)
- seccomp: provides seccomp type definition
- libseccomp: provides utility function that wrappers libseccomp
- forkexec: fork-exec provides mount, unshare, ptrace, seccomp, capset before exec
- memfd: read regular file and creates a seaed memfd for its contents
- unixsocket: send / recv oob msg from a unix socket
- cgroup: creates cgroup directories and collects resource usage / limits
- mount: provides utility function that wrappers mount syscall
- rlimit: provides utility function that defines rlimit syscall
- pipe: provides wrapper to collect all written content through pipe
## Packages
- config: defines arch & language specified trace condition for ptrace runner from UOJ
- container: creates pre-forked container to run programs inside
- runner: interface to run program
- ptrace: wrapper to call forkexec and ptracer
- filehandler: an example implementation of UOJ file set
- unshare: wrapper to call forkexec and unshared namespaces
- ptracer: ptrace tracer and provides syscall trap filter context
## Executable
- runprog: safely run program by unshare / ptrace / pre-forked containers
## Configurations
- config/config.go: all configs toward running specs (similar to UOJ)
## Benchmarks
### ForkExec
```bash
$ go test -bench . -benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/criyle/go-sandbox/pkg/forkexec
BenchmarkSimpleFork-4 12409 996096 ns/op
BenchmarkUnsharePid-4 10000 1065168 ns/op
BenchmarkUnshareUser-4 10000 1061770 ns/op
BenchmarkUnshareUts-4 10000 1056558 ns/op
BenchmarkUnshareCgroup-4 10000 1049446 ns/op
BenchmarkUnshareIpc-4 709 16114052 ns/op
BenchmarkUnshareMount-4 745 16207754 ns/op
BenchmarkUnshareNet-4 3643 3492924 ns/op
BenchmarkFastUnshareMountPivot-4 612 20967318 ns/op
BenchmarkUnshareAll-4 837 14047995 ns/op
BenchmarkUnshareMountPivot-4 488 24198331 ns/op
PASS
ok github.com/criyle/go-sandbox/pkg/forkexec 147.186s
```
### Container
```bash
$ go test -bench . -benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/criyle/go-sandbox/container
BenchmarkContainer-4 5907 2062070 ns/op
PASS
ok github.com/criyle/go-sandbox/container 21.763s
```
### Cgroup
```bash
$ go test -bench . -benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/criyle/go-sandbox/pkg/cgroup
BenchmarkCgroup-4 50283 245094 ns/op
PASS
ok github.com/criyle/go-sandbox/pkg/cgroup 14.744s
```