Now you are be able to capture traffic inside k8s like this:
```
gor --input-raw k8s://namespace/deployment/app:80 --output-http http://replay.com
```
Supported format for filtering required pods:
```
k8s://[namespace/]pod/[pod_name] - k8s://default/pod/nginx-7848d4b86f-5nxz8
k8s://[namespace/]deployment/[deployment_name] - k8s://default/deployment/nginx
k8s://[namespace/]daemonset/[daemonset_name] - k8s://default/daemonset/nginx
k8s://[namespace/]labelSelector/[selector] - k8s://default/labelSelector/app=nginx
k8s://[namespace/]fieldSelector/[selector] - k8s://default/fieldSelector/metadata.name=nginx-7848d4b86f-5nxz8
```
`namespace` is optional, omit to use all namespaces: `k8s://labelSelector/app=replay`
GoReplay designed to be running running as a daemonset (e.g. on each physical k8s node).
See the full guide in here: https://github.com/buger/goreplay/blob/ca8205a5c5d2a1facb00214c78e4120aae6d772d/k8s/README.md
In k8 environment, when listening as daemon set, k8s creates a bunch of virtual interfaces for your traffic with random names like `eni1323`, but in addition it has a classical eth0, or NAT ones like cbr0, which you do not want to listen.
With this option, you now can listen traffic on all virtual interfaces and ignore internal k8s traffic. Example:
`--input-raw-ignore-interface cbr0 --input-raw-ignore-interface eth0 --input-raw-ignore-interface lo`
Also added simple glob pattern `*` for matching multiple interfaces: `--input-raw veth*:80`
Additionally, when you add/remove pod k8s can dynamically add/remove interfaces from the system as well.
Previously, you had to restart the process to notice these changes, now new interfaces detected dynamically, and it automatically starts capture on them. Full example for `GoReplay` to be used as daemon on k8s env:
```
gor --input-raw veth*:80 --output-stdout
```
While running, you will see additional log messages:
```
Found new interface: utun4
Interface: utun4 . BPF Filter: ((tcp dst port 80) and (dst host 10.8.0.2))
```
Added support for capturing virtualized traffic.
## VXLAN
https://en.wikipedia.org/wiki/Virtual_Extensible_LAN
VXLAN implemented as separate engine, which opens UDP socket and awaits traffic.
This approach is made to work with AWS Traffic Mirroring.
In order to enable VXLAN set `--input-raw-engine vxlan`
Example:
```
gor --input-raw :80 --input-raw-engine vxlan --output-stdout`
```
By default, it looks for vxlan traffic on the standard 4789 port, but you can override it with `--input-raw-vxlan-port`.
Additionally, you can allow only specific VNIs using `--input-raw-vxlan-vni`, or disallow by using the same option, but by adding "minus" sign to the value: `--input-raw-vxlan-vni -2`.
Example with all options:
```
gor --input-raw :80 --input-raw-engine vxlan --input-raw-vxlan-vni 1 --input-raw-vxlan-vni 2 --input-raw-vxlan-port 2222 --output-stdout
```
# VLAN
https://en.wikipedia.org/wiki/IEEE_802.1Q
VLAN protocol enabled using `--input-raw-vlan` argument, and you can filter for specific VLAN VIDs using `--input-raw-vlan-vid`. VLAN filtering happens on BPF level.
Example:
```
gor --input-raw :80 --input-raw-vlan --input-raw-vlan-vid 1 --output-stdout`
```
## Notes
Did a refactoring of RAW Input options, so it will be easy to extend in future.
Automatically detect if it is a file (by extension), and do not require specify --input-raw-engine option.
Can also run without port, but in this case, it will assume that port value set to `0` and it will show all records from the file.
Ensure that thread which capture packets as fast as possible.
Packet parsing logic moved to different threads.
Additionally using os.LockOsThread to reduce CPU context switching
Added `--input-raw-allow-incomplete` if you really need it.
Fixed Bug when outpuut binary response not tracked
Additionally fixed bug which prevents Gor from exiting.
Co-authored-by: Leonid Bugaev <leonsbox@gmail.com>
Added a way to capture multiple ports at the same time, with a single listener.
Ports should be separated by comma like this: --input-raw :80,8080,3000
### Reducing CPU context switching and number of goroutines.
Packet capture and packet processing now use only two goroutines which helps to minimize CPU context switches. Spawning too many goroutines is harmful here.
### Optimized packet capture - allocated memory only when required, and only for data which is used
Using ZeroCopy methods from libpcap library to avoid unnecessary allocations. Now memory gets allocated ONLY for the valid packets, and only for the packets which have the data. E.g. no SYN/FIN packets are used now. Additionally we now use `sync.Pool` for re-using packet objects, which helps to re-use already allocated memory.
### Simplification and optimization of request/response detection
There is no SYN/FIN packets anymore etc. Now only packet payload is used to detect start and end of the packet. More over payload detection now does not require generating a total “message” buffer, and works with individual packet payloads.
Message payloads now concatenated from packets only in the end when message is dispatched. Also, before checking if message is complete, added additional check if all received packets in the valid order, e.g. if their SEQ is valid, and no packets are missing.
Reworked chunked encoding validation, and now it does not need expensive operation of re-calculating all the chunks. Now it “trust” that client gives valid chunk body, check if packets are in the right order (e.g. SEQ match), and checks if message ends with the right suffix. All is done with 0 allocations.
Parsing all Headers using `proto.GetHeaders` was proved to be very slow. Now we only parse the headers we need(and do it only once).
Packets gets matched together using ACK, which on high RPS removed chances of duplicating IDs. Additionally, even if packets are received out of order, now it will properly sort them, before dispatching the message.
### Changes in ID generation algorithm
Message ID generation and relations between request and response IDs is fully rewritten. Responses now do not have to lookup for request data in order to get the same ID. ID no rely on the fact that SEQ of the first packet of the response should be the same as ACK of the request. If previously Message ID contained random values, like current timestamp, now it has a consistent algorithm which is based on TCP stream id (SrcPort + DstPort + SrcIP/DstIP) and current ACK/SEQ number (to distinguish multiple messages within the same stream).
### BPF filter optimizations
When tracking response it now uses a more accurate BPF rule to filter only needed traffic.
### Misc
The packet code is now fully moved to tcp/Packet, so packet processing done only once in one place.
TCP output now has a 5 second timeout, and has a proper Close method.
Fully switching to go modules and removing vendoring.
the current UUID generator runs at:
```
BenchmarkMessageUUID 13599342 87.7 ns/op 16 B/op 0 allocs/op
```
with the former version that was running on
```
BenchmarkMessageUUID 2818203 427 ns/op 96 B/op 2 allocs/op
```
it fixes#842#851
// PluginReader is an interface for input plugins
type PluginReader interface {
PluginRead() (msg *Message, err error)
}
// PluginWriter is an interface for output plugins
type PluginWriter interface {
PluginWrite(msg *Message) (n int, err error)
}
The focus here was to **reduce allocation in TCP parser** but speed may have hopeful improved too!
pool no longer use map's key of **string** it uses **uint64**
**Benchmarks** was revamped to be more clear
if you want to compare these results copy the benchmark in tcp/bench_test.go@reduce-allocation to tcp/bench_test.go@master:
**before(master)**:
```
BenchmarkPacketParseAndSort-4 1000000 1006 ns/op 64 B/op 2 allocs/op
BenchmarkMessageParserWithoutHint-4 625 1772309 ns/op 1000 packets/op 419096 B/op 10045 allocs/op
BenchmarkMessageParserWithHint-4 74 14969926 ns/op 1000 chunks/op 1002 packets/op 450992 B/op 10126 allocs/op
```
**After(this branch)**:
```
BenchmarkPacketParseAndSort-4 1267662 941 ns/op 64 B/op 2 allocs/op
BenchmarkMessageParserWithoutHint-4 2256 523474 ns/op 1000 packets/op 243530 B/op 1037 allocs/op
BenchmarkMessageParserWithHint-4 80 13990955 ns/op 1000 chunks/op 1002 packets/op 268609 B/op 1099 allocs/op
```
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive
### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)
### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag
### issues
see linked issues
### tests
- fixed input raw and engine tests
**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
All flags that expect buffer as input i.e. `--output-file-size-limit`, `--output-file-max-size-limit`, `--copy-buffer-size` and `input-raw-buffer-size` can now parse inputs from differents bases and data units like: `10mb`, `10kb`, `100gb`, `18tb`, `11839023`....
data units and bases are case insensitive, the parser accepts only the format of [Go integer literals](https://golang.org/ref/spec#Integer_literals)
Added `—input-raw-buffer-size` - Controls size of the OS buffer (in
bytes) which holds packets until they dispatched. Default value depends
by system: in Linux around 2MB. If you see big package drop, increase
this value.
Additionally snaplen (max number of bytes being read for each packet)
now dynamically set based on interface MTU + max header size. In most
situations it should reduce package drop, because each packet will
consume less space in buffer.
Previously latency calcualted as Response.End - Request.Start
Where both End and Start is a last and first packets
This calcualtion is wrong, because it is total roundtrip
Correct server latency will be Response.End - Request.End
In addition added new `--input-raw-timestamp-type` option
which allows choose more precise packet timestamp source (if available).
Can be useful in case of non standard network interfaces when like
tunnels or SPAN ports, when IP of interface does not match with IP of
packet.
Can be used to read traffic from multiple ports and interfaces at once
as well.
