In k8 environment, when listening as daemon set, k8s creates a bunch of virtual interfaces for your traffic with random names like `eni1323`, but in addition it has a classical eth0, or NAT ones like cbr0, which you do not want to listen.
With this option, you now can listen traffic on all virtual interfaces and ignore internal k8s traffic. Example:
`--input-raw-ignore-interface cbr0 --input-raw-ignore-interface eth0 --input-raw-ignore-interface lo`
Also added simple glob pattern `*` for matching multiple interfaces: `--input-raw veth*:80`
Additionally, when you add/remove pod k8s can dynamically add/remove interfaces from the system as well.
Previously, you had to restart the process to notice these changes, now new interfaces detected dynamically, and it automatically starts capture on them. Full example for `GoReplay` to be used as daemon on k8s env:
```
gor --input-raw veth*:80 --output-stdout
```
While running, you will see additional log messages:
```
Found new interface: utun4
Interface: utun4 . BPF Filter: ((tcp dst port 80) and (dst host 10.8.0.2))
```
Added support for capturing virtualized traffic.
## VXLAN
https://en.wikipedia.org/wiki/Virtual_Extensible_LAN
VXLAN implemented as separate engine, which opens UDP socket and awaits traffic.
This approach is made to work with AWS Traffic Mirroring.
In order to enable VXLAN set `--input-raw-engine vxlan`
Example:
```
gor --input-raw :80 --input-raw-engine vxlan --output-stdout`
```
By default, it looks for vxlan traffic on the standard 4789 port, but you can override it with `--input-raw-vxlan-port`.
Additionally, you can allow only specific VNIs using `--input-raw-vxlan-vni`, or disallow by using the same option, but by adding "minus" sign to the value: `--input-raw-vxlan-vni -2`.
Example with all options:
```
gor --input-raw :80 --input-raw-engine vxlan --input-raw-vxlan-vni 1 --input-raw-vxlan-vni 2 --input-raw-vxlan-port 2222 --output-stdout
```
# VLAN
https://en.wikipedia.org/wiki/IEEE_802.1Q
VLAN protocol enabled using `--input-raw-vlan` argument, and you can filter for specific VLAN VIDs using `--input-raw-vlan-vid`. VLAN filtering happens on BPF level.
Example:
```
gor --input-raw :80 --input-raw-vlan --input-raw-vlan-vid 1 --output-stdout`
```
## Notes
Did a refactoring of RAW Input options, so it will be easy to extend in future.
In some low traffic cases you can have cases when time between request minutes.
Additionally increasing speed can be not an option.
Now you can "skip" this pauses, by seetting max wait time
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Input file now pre-reads N requests, sort them by timestamp and emit on demand.
You can control read depth using --input-file-read-depth which is 100 by default.
It makes implementaiton faster, and it fix various issues when due to concurrenccy, or another issues requests gets addeed out of order.
Added `--input-raw-allow-incomplete` if you really need it.
Fixed Bug when outpuut binary response not tracked
Additionally fixed bug which prevents Gor from exiting.
// PluginReader is an interface for input plugins
type PluginReader interface {
PluginRead() (msg *Message, err error)
}
// PluginWriter is an interface for output plugins
type PluginWriter interface {
PluginWrite(msg *Message) (n int, err error)
}
Allow to configure the output to ignore TLS certificates that doesn't match the hostname.
This is necessary when connecting using an IP address instead of a domain or using a self signed certificate.
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive
### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)
### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag
### issues
see linked issues
### tests
- fixed input raw and engine tests
**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
the major purpose of this PR is to update modules to their recent **minor version**.
this is helpful in terms of having update functionalities and fixing some issues caused by non-up-to-date modules.
All flags that expect buffer as input i.e. `--output-file-size-limit`, `--output-file-max-size-limit`, `--copy-buffer-size` and `input-raw-buffer-size` can now parse inputs from differents bases and data units like: `10mb`, `10kb`, `100gb`, `18tb`, `11839023`....
data units and bases are case insensitive, the parser accepts only the format of [Go integer literals](https://golang.org/ref/spec#Integer_literals)
This provides the ability to have a maximum number of workers, yet still allow scaling down. The scaling down is important to stop timeouts happening with under-utilised workers.
Note, although the code has workers_min, workers_max, the settings option --output-http-workers has remained and becomes the "max" value. I did this to hopefully not break too much for others. The min will default to 1, so users will see dynamic scaling unless they also set min to be the same. This will probably benefit them as they will avoid potential timeouts when the traffic reduces. But something to be aware of.
Added new `--http-pprof` option, example: "--http-pprof :8181"
It starts web server on given address, and expose special /debug/pprof
endpoint
See https://golang.org/pkg/net/http/pprof/
Added `—input-raw-buffer-size` - Controls size of the OS buffer (in
bytes) which holds packets until they dispatched. Default value depends
by system: in Linux around 2MB. If you see big package drop, increase
this value.
Additionally snaplen (max number of bytes being read for each packet)
now dynamically set based on interface MTU + max header size. In most
situations it should reduce package drop, because each packet will
consume less space in buffer.
Previously latency calcualted as Response.End - Request.Start
Where both End and Start is a last and first packets
This calcualtion is wrong, because it is total roundtrip
Correct server latency will be Response.End - Request.End
In addition added new `--input-raw-timestamp-type` option
which allows choose more precise packet timestamp source (if available).
