This PR does two primary things:
- ensures that HasFullPayload returns false if the packet doesnt start with a valid request or response header. This was necessary because a chunked response would return true from HasFullPayload if it got contiguous packets including the last packet (with the trailer) before it got the first packet (with the header).
- When the request payload is chunked across multiple packets, only correct for 100-Continue responses once. In requests with > 2 packets, the Ack number is incremented for each packet in the message, sometimes resulting in packets not being correlated with each other. This ensures that the message is corrected once and only once.
Fix message size check in `TestMessageTimeoutReached`. Since the message parser has parsed two packets of size 63 << 10, then the message size should be 63 << 11
In some cases recorded files contain small amount malformed records.
The root cause is not clear, maybe multiple processes writing to the same file.
This change ensuer that file can be replayed, and malformed records will be ingored (with meaningful debug message showing exact line in the file with issue).
Additionally `--input-file-dry-run` mode was speed up a bit, since there were a few of Sleep statements which were not ignored when dry run is executed.
This PR contains multiple fixes:
- Handle TCP padding (zeroes at the end of TCP payload), and do not treat it as a body
- Handle requests with "Expect: 100-Continue" - the ones which require confirmation from the server, before sending the body
- Fix muti-packet headers parsing, if "truncated" header starts with malformed header format
- Fix replay of pcap files (Ignore Stats method since it is not supported)
- Fix output file chunk size detection
1. move the `isDevice(l.host, pi)` to be first, as no need to iterate on all nics if it returns `true`
2. first compare by name, as same nics will have same names
3. if not found by name, compare by ips.
the bug was the `strings.HasPrefix`
2 different nics with ipv6:
```
#nic1 ip: f1234::55
#nic2 ip: f1234::55::66::66
```
so because of the `strings.HasPrefix` it was evaluated as the name nics. but they are not.
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e
Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.
Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.
Fix#989
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e
Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.
Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.
Fix#989
If packet processign is not fast enough, packet capture may miss packets
Now it use 10 goroutines, which distribute work based on the ephemeral port number
This should fix#973 . Tested with the following command:
```bash
gor --input-raw 127.0.0.1:8080 \
--output-file '%Y-%m-%d.log' \
--output-file-queue-limit 0 \
--output-file-size-limit 1mb
```
Ensure that thread which capture packets as fast as possible.
Packet parsing logic moved to different threads.
Additionally using os.LockOsThread to reduce CPU context switching
# Reading in advance
Input file now pre-reads N requests (and keeps this buffer on the same length), sort them by timestamp and emit on demand.
You can control read depth using `--input-file-read-depth` which is 100 by default.
It makes implementation faster, and it fixes various issues when due to concurrency, or another issues requests gets added out of order.
Setting too big depth will mean that memory consumption will be bigger, since it will need store request in memory.
# Max wait
In some low traffic cases, you can have cases when time between request minutes.
Increasing speed can be not an option.
Now you can "skip" this pauses, by setting max wait time. Example: `--input-raw-max-wait 1s`.
# Dry-run mode
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Example report:
```
Records found: 258
Files processed: 1
Bytes processed: 208343
Max wait: 1h26m3.193159s
Min wait: 15µs
First wait: 1.174ms
It will take `1h41m44.614806s` to replay at current speed.
```
Applying options like input-raw-max-wait, setting speed, or read depth affecting dry-run mode as well.
# Misc
Fix issue with re-using buffers for messages. On concurrency, buffers gets re-used, and you get corrupted data.
In some low traffic cases you can have cases when time between request minutes.
Additionally increasing speed can be not an option.
Now you can "skip" this pauses, by seetting max wait time
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Input file now pre-reads N requests, sort them by timestamp and emit on demand.
You can control read depth using --input-file-read-depth which is 100 by default.
It makes implementaiton faster, and it fix various issues when due to concurrenccy, or another issues requests gets addeed out of order.
Windows having issues with generating timestamps, so adding application level timestamp generation
Made small refactoring to move "accurate-enough" time to own package.
In some rare cases, requests can be recorded out of order.
It was causing replay from file significanty slodown when such issue met.
Now it replays them but without adding sleep.
When you use S3 and write from multiple machines, you need to ensure that file will have unique name for each machine.
This PR adds special %i function which you can use in file templates, which will add some random process uniq string to the file.
Leonid Bugaev