updated readme & enforced sync

This commit is contained in:
criyle 2019-08-12 01:36:15 -07:00
parent 60ff0c39fd
commit 4452ebd803
6 changed files with 143 additions and 87 deletions

135
README.md
View File

@ -1,76 +1,103 @@
## go-judger
# go-judger
Under developing.
Original goal is to reimplement [uoj-judger/run_program](https://github.com/vfleaking/uoj) in GO language using [libseccomp](https://github.com/seccomp/libseccomp-golang). As technology grows, implements new technologies such as Linux namespace & cgroup.
Goal is to reimplement [uoj-judger/run_program](https://github.com/vfleaking/uoj) in GO language using [libseccomp](https://github.com/seccomp/libseccomp-golang).
## Install
Install:
- install go compiler: `apt install golang-go`
- install libseccomp-dev: `apt install libseccomp-dev`
- install: `go install github.com/criyle/go-judger/...`
- install go compiler: `apt install golang-go`
- install libseccomp-dev: `apt install libseccomp-dev`
- install: `go install github.com/criyle/go-judger/...`
## Technologies
Features (same as uoj-judger/run_program):
### libseccomp + ptrace (improved UOJ sandbox)
1. Restricted computing resource: Time & Memory (Stack) & Output
2. Restricted syscall access (by libseccomp & ptrace)
3. Restricted file access (read & write & access & exec)
1. Restricted computing resource by POSIX rlimit: Time & Memory (Stack) & Output
2. Restricted syscall access (by libseccomp & ptrace)
3. Restricted file access (read & write & access & exec). Evaluated by UOJ FileSet
New Features:
Improvements:
1. Percise resource limits (s -> ms, mb -> kb)
2. More architectures (arm32, arm64, x86)
3. Allow multiple traced programs in different threads
4. Allow pipes as input / output files
5. Use Linux Namespace to isolate file access (elimilate ptrace)
6. Use Linux Control Groups to limit & acct CPU & memory (elimilate wait4.rusage)
7. Container tech with execveat memfd, sethostname, setdomainname
8. Pre-fork container deamons to run programs inside
1. Percise resource limits (s -> ms, mb -> kb)
2. More architectures (arm32, arm64, x86)
3. Allow multiple traced programs in different threads
4. Allow pipes as input / output files
Default file access action:
Default file access syscall check:
- check file read / write: `open`, `openat`
- check file read: `readlink`, `readlinkat`
- check file write: `unlink`, `unlinkat`, `chmod`, `rename`
- check file access: `stat`, `lstat`, `access`, `faccessat`
- check file exec: `execveat`
- check file read / write: `open`, `openat`
- check file read: `readlink`, `readlinkat`
- check file write: `unlink`, `unlinkat`, `chmod`, `rename`
- check file access: `stat`, `lstat`, `access`, `faccessat`
- check file exec: `execveat`
Packages:
### linux namespace + cgroup
- seccomp: provides utility function that wrappers libseccomp
- forkexec: fork-exec provides mount, unshare, ptrace, seccomp, capset before exec
- memfd: read regular file and creates a seaed memfd for its contents
- unixsocket: send / recv oob msg from a unix socket
- cgroup: creates cgroup directories and collects resource usage / limits
- deamon: creates pre-forked container to run programs inside
- tracer: ptrace tracer and provides syscall trap filter context
- runprogram: wrapper to call forkexec and trecer
- rununshared: wrapper to call forkexec and unshared namespaces
- runconfig: defines arch & language specified trace condition for seccomp and ptrace
- types: general runtime specs
- mount: provides utility function that wrappers mount syscall
- rlimit: provides utility function that defines rlimit syscall
- specs: provides general res / result data structures
1. Unshare & bind mount rootfs based on hostfs (elimilated ptrace)
2. Use Linux Control Groups to limit & acct CPU & memory (elimilate wait4.rusage)
3. Container tech with execveat memfd, sethostname, setdomainname
Executable:
### pre-forked container
- run_program: safely run program by unshare / ptrace / pre-forked containers
1. Pre-fork container deamons to run programs inside
2. Unix socket to pass fd inside / outside
Configuations:
## Packages
- run_program/config.go: all configs toward running specs
- seccomp: provides utility function that wrappers libseccomp
- forkexec: fork-exec provides mount, unshare, ptrace, seccomp, capset before exec
- memfd: read regular file and creates a seaed memfd for its contents
- unixsocket: send / recv oob msg from a unix socket
- cgroup: creates cgroup directories and collects resource usage / limits
- deamon: creates pre-forked container to run programs inside
- tracer: ptrace tracer and provides syscall trap filter context
- runprogram: wrapper to call forkexec and trecer
- rununshared: wrapper to call forkexec and unshared namespaces
- runconfig: defines arch & language specified trace condition for seccomp and ptrace
- types: general runtime specs
- mount: provides utility function that wrappers mount syscall
- rlimit: provides utility function that defines rlimit syscall
- specs: provides general res / result data structures
Benchmarks (docker desktop amd64 / native arm64):
## Executable
- 1ms / 2ms: fork, unshare pid / user / cgroup
- 4ms / 8ms: run inside pre-forked container
- 50ms / 25ms: unshare ipc / mount
- 100ms / 44ms: unshare pid & user & cgroup & mount & pivot root
- 400ms / 63ms: unshare net
- 800ms / 170ms: unshare all
- 880ms / 170ms: unshare all & pivot root
- run_program: safely run program by unshare / ptrace / pre-forked containers
## Configuations
- run_program/config.go: all configs toward running specs
## Benchmarks (docker desktop amd64 / native arm64)
- 1ms / 2ms: fork, unshare pid / user / cgroup
- 4ms / 8ms: run inside pre-forked container
- 50ms / 25ms: unshare ipc / mount
- 100ms / 44ms: unshare pid & user & cgroup & mount & pivot root
- 400ms / 63ms: unshare net
- 800ms / 170ms: unshare all
- 880ms / 170ms: unshare all & pivot root
It seems unshare net or ipc takes time, maybe limits action by seccomp instead.
Pre-forked container also saves time for container creation / cleanup.
TODO:
```bash
$ go test -bench . -benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/criyle/go-judger/forkexec
BenchmarkSimpleFork-4 10000 1106064 ns/op
BenchmarkUnsharePid-4 10000 1367824 ns/op
BenchmarkUnshareUser-4 10000 1311523 ns/op
BenchmarkUnshareUts-4 10000 1140427 ns/op
BenchmarkUnshareCgroup-4 10000 1112713 ns/op
BenchmarkUnshareIpc-4 300 58730786 ns/op
BenchmarkUnshareMount-4 300 55540758 ns/op
BenchmarkUnshareNet-4 100 396957720 ns/op
BenchmarkFastUnshareMountPivot-4 100 114364585 ns/op
BenchmarkUnshareAll-4 100 851014031 ns/op
BenchmarkUnshareMountPivot-4 20 901204445 ns/op
PASS
ok github.com/criyle/go-judger/forkexec 262.112s
```
## TODO

View File

@ -20,10 +20,7 @@ import (
func ContainerInit() (err error) {
// noop if self is not container init process
// Notice: docker init is also 1, additional check for args[1] == init
if os.Getpid() != 1 {
return nil
}
if len(os.Args) != 2 || os.Args[1] != initArg {
if os.Getpid() != 1 || len(os.Args) != 2 || os.Args[1] != initArg {
return nil
}
@ -255,8 +252,7 @@ func recvCmd(s *unixsocket.Socket) (*Cmd, *unixsocket.Msg, error) {
if err != nil {
return nil, nil, fmt.Errorf("failed RecvMsg(%v)", err)
}
dec := gob.NewDecoder(bytes.NewReader(buffer[:n]))
if err := dec.Decode(&cmd); err != nil {
if err := gob.NewDecoder(bytes.NewReader(buffer[:n])).Decode(&cmd); err != nil {
return nil, nil, fmt.Errorf("failed to decode(%v)", err)
}
return &cmd, msg, nil
@ -264,8 +260,7 @@ func recvCmd(s *unixsocket.Socket) (*Cmd, *unixsocket.Msg, error) {
func sendReply(s *unixsocket.Socket, reply *Reply, msg *unixsocket.Msg) error {
var buffer bytes.Buffer
enc := gob.NewEncoder(&buffer)
if err := enc.Encode(reply); err != nil {
if err := gob.NewEncoder(&buffer).Encode(reply); err != nil {
return err
}
if err := s.SendMsg(buffer.Bytes(), msg); err != nil {

View File

@ -107,8 +107,7 @@ func (m *Master) recvReply() (*Reply, *unixsocket.Msg, error) {
if err != nil {
return nil, nil, fmt.Errorf("failed to RecvMsg(%v)", err)
}
dec := gob.NewDecoder(bytes.NewReader(buff[:n]))
if err := dec.Decode(&reply); err != nil {
if err := gob.NewDecoder(bytes.NewReader(buff[:n])).Decode(&reply); err != nil {
return nil, nil, fmt.Errorf("failed to decode(%v)", err)
}
return &reply, msg, nil
@ -116,8 +115,7 @@ func (m *Master) recvReply() (*Reply, *unixsocket.Msg, error) {
func (m *Master) sendCmd(cmd *Cmd, msg *unixsocket.Msg) error {
var buff bytes.Buffer
enc := gob.NewEncoder(&buff)
if err := enc.Encode(cmd); err != nil {
if err := gob.NewEncoder(&buff).Encode(cmd); err != nil {
return fmt.Errorf("failed to encode(%v)", err)
}
if err := m.socket.SendMsg(buff.Bytes(), msg); err != nil {

View File

@ -86,12 +86,6 @@ func (r *Runner) Start() (int, error) {
// similar to exec_linux, avoid side effect by shuffling around
fd, nextfd := prepareFds(r.Files)
pipe := p2[1]
if nextfd <= pipe {
nextfd = pipe + 1
}
if nextfd <= int(r.ExecFile) {
nextfd = int(r.ExecFile) + 1
}
// Acquire the fork lock so that no other threads
// create new fds that are not yet close-on-exec
@ -109,11 +103,15 @@ func (r *Runner) Start() (int, error) {
afterFork()
syscall.ForkLock.Unlock()
// clone syscall failed
if err1 != 0 {
return int(pid), syscall.Errno(err1)
}
// synchronize with child for uid / gid map
if unshareUser {
unix.Close(p[0])
err = writeIDMaps(int(pid))
if err != nil {
if err = writeIDMaps(int(pid)); err != nil {
err2 = err.(syscall.Errno)
}
syscall.RawSyscall(syscall.SYS_WRITE, uintptr(p[1]), uintptr(unsafe.Pointer(&err2)), uintptr(unsafe.Sizeof(err2)))
@ -122,24 +120,46 @@ func (r *Runner) Start() (int, error) {
// sync with child
unix.Close(p2[1])
_, _, err1 = syscall.RawSyscall(syscall.SYS_READ, uintptr(p2[0]), uintptr(unsafe.Pointer(&err2)), uintptr(unsafe.Sizeof(err2)))
if err1 != 0 {
r1, _, err1 = syscall.RawSyscall(syscall.SYS_READ, uintptr(p2[0]), uintptr(unsafe.Pointer(&err2)), uintptr(unsafe.Sizeof(err2)))
// child returned error code
if r1 != unsafe.Sizeof(err2) || err2 != 0 || err1 != 0 {
unix.Close(p2[0])
return int(pid), syscall.Errno(err1)
if r1 == unsafe.Sizeof(err2) {
err = syscall.Errno(err2)
}
if err == nil {
err = syscall.EPIPE
}
handleChildFailed(pid)
return 0, err
}
if r.SyncFunc != nil {
err = r.SyncFunc(int(pid))
}
// if syncfunc return error, then fail child immediately
if err == nil {
err1 = 0
syscall.RawSyscall(syscall.SYS_WRITE, uintptr(p2[0]), uintptr(unsafe.Pointer(&err1)), uintptr(unsafe.Sizeof((err1))))
} else {
unix.Close(p2[0])
handleChildFailed(pid)
return 0, err
}
// if read anything mean child failed after sync (close_on_exec so it should not block)
r1, _, err1 = syscall.RawSyscall(syscall.SYS_READ, uintptr(p2[0]), uintptr(unsafe.Pointer(&err2)), uintptr(unsafe.Sizeof(err2)))
unix.Close(p2[0])
if err1 != 0 {
return int(pid), syscall.Errno(err1)
if r1 != 0 || err1 != 0 {
if r1 == unsafe.Sizeof(err2) {
err = syscall.Errno(err2)
}
if err == nil {
err = syscall.EPIPE
}
handleChildFailed(pid)
return 0, err
}
return int(pid), nil
}
@ -199,6 +219,10 @@ func (r *Runner) Start() (int, error) {
nextfd++
}
for i := 0; i < len(fd); i++ {
// Avoid fd rewrite
for nextfd == i || (r.ExecFile > 0 && nextfd == int(r.ExecFile)) {
nextfd++
}
if fd[i] >= 0 && fd[i] < int(i) {
_, _, err1 = syscall.RawSyscall(syscall.SYS_DUP3, uintptr(fd[i]), uintptr(nextfd), syscall.O_CLOEXEC)
if err1 != 0 {
@ -453,7 +477,21 @@ func (r *Runner) Start() (int, error) {
}
childerror:
syscall.RawSyscall(syscall.SYS_EXIT, uintptr(err1+err2), 0, 0)
// send error code on pipe
syscall.RawSyscall(unix.SYS_WRITE, uintptr(pipe), uintptr(unsafe.Pointer(&err1)), unsafe.Sizeof(err1))
for {
syscall.RawSyscall(syscall.SYS_EXIT, uintptr(err1+err2), 0, 0)
}
// cannot reach this point
panic("cannot reach")
}
func handleChildFailed(pid uintptr) {
var wstatus syscall.WaitStatus
// make sure not blocked
unix.Kill(int(pid), syscall.SIGKILL)
// child failed; wait for it to exit, to make sure the zombies don't accumulate
_, err3 := syscall.Wait4(int(pid), &wstatus, 0, nil)
for err3 == syscall.EINTR {
_, err3 = syscall.Wait4(int(pid), &wstatus, 0, nil)
}
}

2
go.mod
View File

@ -4,5 +4,5 @@ go 1.12
require (
github.com/seccomp/libseccomp-golang v0.9.1
golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa
)

6
go.sum
View File

@ -1,6 +1,4 @@
github.com/seccomp/libseccomp-golang v0.9.1 h1:NJjM5DNFOs0s3kYE1WUOr6G8V97sdt46rlXTMfXGWBo=
github.com/seccomp/libseccomp-golang v0.9.1/go.mod h1:GbW5+tmTXfcxTToHLXlScSlAvWlF4P2Ca7zGrPiEpWo=
golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k=
golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 h1:LepdCS8Gf/MVejFIt8lsiexZATdoGVyp5bcyS+rYoUI=
golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa h1:KIDDMLT1O0Nr7TSxp8xM5tJcdn8tgyAONntO829og1M=
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=