Scratch doesn't have root certificates by default, which means forwarding to a HTTPS endpoint doesn't work without skipping verification. To fix this, we can copy the certificate bundle into scratch from the alpine image.
I've also bumped the alpine version up to 3.16, and fixed the wget.
This has fixed my issues, but let me know if you think anything looks wrong here!
The `copySlice` function inside `tcp_packet.go` wouldn't copy elements from the `from` slices to the `to` slice, even when the `to` slice has "space" to spare due to its `cap` being larger than the `totalLen`:
```go
func copySlice(to []byte, skip int, from ...[]byte) ([]byte, int) {
var totalLen int
for _, s := range from {
totalLen += len(s)
}
totalLen += skip
if cap(to) < totalLen {
diff := totalLen - cap(to)
to = append(to, make([]byte, diff)...)
}
for _, s := range from {
skip += copy(to[skip:], s)
}
return to, skip
}
```
This is caused because Go's `copy` function used in `copySlice` will copy a number of elements ["which will be the minimum of len(src) and len(dst)."](https://pkg.go.dev/builtin#copy). For the built-in copy function to copy elements into a slice, the destination slots must be initialized for this slice, not just allocated in the underlying array. In other words, `len` must be used instead of `cap` to allow `copy` to work properly.
This PR closes#1095, which is an example of the effects of this issue: when mirroring packets using VXLAN, the raw payloads obtained using the vxlan engine can be large due to encapsulation. This has the effect of giving the `tmp` slice a large `cap` when `PacketData()` is called inside `tcp_message.go`. Then, the `to` slice is never resized in `copySlice` and only the first packet is read, without the rest (e.g. no response body, only headers are read).
Now you are be able to capture traffic inside k8s like this:
```
gor --input-raw k8s://namespace/deployment/app:80 --output-http http://replay.com
```
Supported format for filtering required pods:
```
k8s://[namespace/]pod/[pod_name] - k8s://default/pod/nginx-7848d4b86f-5nxz8
k8s://[namespace/]deployment/[deployment_name] - k8s://default/deployment/nginx
k8s://[namespace/]daemonset/[daemonset_name] - k8s://default/daemonset/nginx
k8s://[namespace/]labelSelector/[selector] - k8s://default/labelSelector/app=nginx
k8s://[namespace/]fieldSelector/[selector] - k8s://default/fieldSelector/metadata.name=nginx-7848d4b86f-5nxz8
```
`namespace` is optional, omit to use all namespaces: `k8s://labelSelector/app=replay`
GoReplay designed to be running running as a daemonset (e.g. on each physical k8s node).
See the full guide in here: https://github.com/buger/goreplay/blob/ca8205a5c5d2a1facb00214c78e4120aae6d772d/k8s/README.md
In k8 environment, when listening as daemon set, k8s creates a bunch of virtual interfaces for your traffic with random names like `eni1323`, but in addition it has a classical eth0, or NAT ones like cbr0, which you do not want to listen.
With this option, you now can listen traffic on all virtual interfaces and ignore internal k8s traffic. Example:
`--input-raw-ignore-interface cbr0 --input-raw-ignore-interface eth0 --input-raw-ignore-interface lo`
Also added simple glob pattern `*` for matching multiple interfaces: `--input-raw veth*:80`
Additionally, when you add/remove pod k8s can dynamically add/remove interfaces from the system as well.
Previously, you had to restart the process to notice these changes, now new interfaces detected dynamically, and it automatically starts capture on them. Full example for `GoReplay` to be used as daemon on k8s env:
```
gor --input-raw veth*:80 --output-stdout
```
While running, you will see additional log messages:
```
Found new interface: utun4
Interface: utun4 . BPF Filter: ((tcp dst port 80) and (dst host 10.8.0.2))
```
Added support for capturing virtualized traffic.
## VXLAN
https://en.wikipedia.org/wiki/Virtual_Extensible_LAN
VXLAN implemented as separate engine, which opens UDP socket and awaits traffic.
This approach is made to work with AWS Traffic Mirroring.
In order to enable VXLAN set `--input-raw-engine vxlan`
Example:
```
gor --input-raw :80 --input-raw-engine vxlan --output-stdout`
```
By default, it looks for vxlan traffic on the standard 4789 port, but you can override it with `--input-raw-vxlan-port`.
Additionally, you can allow only specific VNIs using `--input-raw-vxlan-vni`, or disallow by using the same option, but by adding "minus" sign to the value: `--input-raw-vxlan-vni -2`.
Example with all options:
```
gor --input-raw :80 --input-raw-engine vxlan --input-raw-vxlan-vni 1 --input-raw-vxlan-vni 2 --input-raw-vxlan-port 2222 --output-stdout
```
# VLAN
https://en.wikipedia.org/wiki/IEEE_802.1Q
VLAN protocol enabled using `--input-raw-vlan` argument, and you can filter for specific VLAN VIDs using `--input-raw-vlan-vid`. VLAN filtering happens on BPF level.
Example:
```
gor --input-raw :80 --input-raw-vlan --input-raw-vlan-vid 1 --output-stdout`
```
## Notes
Did a refactoring of RAW Input options, so it will be easy to extend in future.
Automatically detect if it is a file (by extension), and do not require specify --input-raw-engine option.
Can also run without port, but in this case, it will assume that port value set to `0` and it will show all records from the file.
This PR does two primary things:
- ensures that HasFullPayload returns false if the packet doesnt start with a valid request or response header. This was necessary because a chunked response would return true from HasFullPayload if it got contiguous packets including the last packet (with the trailer) before it got the first packet (with the header).
- When the request payload is chunked across multiple packets, only correct for 100-Continue responses once. In requests with > 2 packets, the Ack number is incremented for each packet in the message, sometimes resulting in packets not being correlated with each other. This ensures that the message is corrected once and only once.
In some cases recorded files contain small amount malformed records.
The root cause is not clear, maybe multiple processes writing to the same file.
This change ensuer that file can be replayed, and malformed records will be ingored (with meaningful debug message showing exact line in the file with issue).
Additionally `--input-file-dry-run` mode was speed up a bit, since there were a few of Sleep statements which were not ignored when dry run is executed.
Fix message size check in `TestMessageTimeoutReached`. Since the message parser has parsed two packets of size 63 << 10, then the message size should be 63 << 11
This PR contains multiple fixes:
- Handle TCP padding (zeroes at the end of TCP payload), and do not treat it as a body
- Handle requests with "Expect: 100-Continue" - the ones which require confirmation from the server, before sending the body
- Fix muti-packet headers parsing, if "truncated" header starts with malformed header format
- Fix replay of pcap files (Ignore Stats method since it is not supported)
- Fix output file chunk size detection
1. move the `isDevice(l.host, pi)` to be first, as no need to iterate on all nics if it returns `true`
2. first compare by name, as same nics will have same names
3. if not found by name, compare by ips.
the bug was the `strings.HasPrefix`
2 different nics with ipv6:
```
#nic1 ip: f1234::55
#nic2 ip: f1234::55::66::66
```
so because of the `strings.HasPrefix` it was evaluated as the name nics. but they are not.
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e
Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.
Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.
Fix#989
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e
Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.
Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.
Fix#989
If packet processign is not fast enough, packet capture may miss packets
Now it use 10 goroutines, which distribute work based on the ephemeral port number
This should fix#973 . Tested with the following command:
```bash
gor --input-raw 127.0.0.1:8080 \
--output-file '%Y-%m-%d.log' \
--output-file-queue-limit 0 \
--output-file-size-limit 1mb
```
Ensure that thread which capture packets as fast as possible.
Packet parsing logic moved to different threads.
Additionally using os.LockOsThread to reduce CPU context switching
