# Reading in advance
Input file now pre-reads N requests (and keeps this buffer on the same length), sort them by timestamp and emit on demand.
You can control read depth using `--input-file-read-depth` which is 100 by default.
It makes implementation faster, and it fixes various issues when due to concurrency, or another issues requests gets added out of order.
Setting too big depth will mean that memory consumption will be bigger, since it will need store request in memory.
# Max wait
In some low traffic cases, you can have cases when time between request minutes.
Increasing speed can be not an option.
Now you can "skip" this pauses, by setting max wait time. Example: `--input-raw-max-wait 1s`.
# Dry-run mode
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Example report:
```
Records found: 258
Files processed: 1
Bytes processed: 208343
Max wait: 1h26m3.193159s
Min wait: 15µs
First wait: 1.174ms
It will take `1h41m44.614806s` to replay at current speed.
```
Applying options like input-raw-max-wait, setting speed, or read depth affecting dry-run mode as well.
# Misc
Fix issue with re-using buffers for messages. On concurrency, buffers gets re-used, and you get corrupted data.
In some low traffic cases you can have cases when time between request minutes.
Additionally increasing speed can be not an option.
Now you can "skip" this pauses, by seetting max wait time
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Input file now pre-reads N requests, sort them by timestamp and emit on demand.
You can control read depth using --input-file-read-depth which is 100 by default.
It makes implementaiton faster, and it fix various issues when due to concurrenccy, or another issues requests gets addeed out of order.
Windows having issues with generating timestamps, so adding application level timestamp generation
Made small refactoring to move "accurate-enough" time to own package.
In some rare cases, requests can be recorded out of order.
It was causing replay from file significanty slodown when such issue met.
Now it replays them but without adding sleep.
When you use S3 and write from multiple machines, you need to ensure that file will have unique name for each machine.
This PR adds special %i function which you can use in file templates, which will add some random process uniq string to the file.
Added `--input-raw-allow-incomplete` if you really need it.
Fixed Bug when outpuut binary response not tracked
Additionally fixed bug which prevents Gor from exiting.
Do not use sync.Pool, use channel based impementation instead. Sync pool does not work for long lived objects, and trigger frequent GC.
Because of MUCH more efficient memory utilization, GOGC now set to 500, which significantly improve CPU as wel.
The cause was not ZeroCopy but wrong SetTimeout (block cause a lot of overhead).
Packet processing previously used channels, but now, with ring buffer, using select was causing issue. Adding `default` clause fixed the issue.
It is a high performant alternative to libpcap engine.
Only Linux supported.
Performance gain can be up to 50% depending on traffic type.
Can be enabled using:
`--input-raw-engine af_packet`
Issues is that Go built-in net.Interfaces function in newer Windows versions return wrong interface names, which libpcap can't consume.
Now we use pcap.FindDevices instead of net.Interfaces.
See this Article for deep understanding of the issue https://haydz.github.io/2020/07/06/Go-Windows-NIC.html
Additionally, found a bug causing big memory allocations, for large requests, when we perform check if messages finished or not.
Because of this bug chunked body encoding check was not working properly.
Was not caught in tests, because test was working on packet array level, and this issue happens when dealing with TCP message object.
Additionally, added a small fix for windows Makefile task, it now generates proper file name.
For a long time there were no official binaries for the windows platform.
One of the reasons is the complexities of the build toolchain. Not only CGO is required, but also installing the needed libraries and header files, not talking about mingw and etc.
2 weeks ago [golang-crossbuild](https://github.com/elastic/golang-crossbuild) project added native support for Libpcap based applications.
Windows support is based on the WinPcap which is a bit (a lot) outdated, BUT, since we depend only on its interface, it is still possible to use projects like npcap https://nmap.org/npcap/#download.
Npcap needs be installed with WinPcap compatibe mode (checkbox during installation)
It is also possibe install it in silent mode like this: `npcap-0.86.exe /S /winpcap_mode=yes`
After PR merge, will be updated related documentation.
Additionally fix dependency on "unix" package (apparently it can be totally replaced using universal syscall package)
For a long time there were no official binaries for the windows platform.
One of the reasons is the complexities of the build toolchain. Not only CGO is required, but also installing the needed libraries and header files, not talking about mingw and etc.
2 weeks ago [golang-crossbuild](https://github.com/elastic/golang-crossbuild) project added native support for Libpcap based applications.
Windows support is based on the WinPcap which is a bit (a lot) outdated, BUT, since we depend only on its interface, it is still possible to use projects like npcap https://nmap.org/npcap/#download.
Npcap needs be installed with WinPcap compatibe mode (checkbox during installation)
It is also possibe install it in silent mode like this: `npcap-0.86.exe /S /winpcap_mode=yes`
After PR merge, will be updated related documentation.
Additionally fix dependency on "unix" package (apparently it can be totally replaced using universal syscall package)
Added new `--http-pprof` option, example: "--http-pprof :8181"
It starts web server on given address, and expose special /debug/pprof
endpoint
See https://golang.org/pkg/net/http/pprof/
Added `—input-raw-buffer-size` - Controls size of the OS buffer (in
bytes) which holds packets until they dispatched. Default value depends
by system: in Linux around 2MB. If you see big package drop, increase
this value.
Additionally snaplen (max number of bytes being read for each packet)
now dynamically set based on interface MTU + max header size. In most
situations it should reduce package drop, because each packet will
consume less space in buffer.
Previously latency calcualted as Response.End - Request.Start
Where both End and Start is a last and first packets
This calcualtion is wrong, because it is total roundtrip
Correct server latency will be Response.End - Request.End
In addition added new `--input-raw-timestamp-type` option
which allows choose more precise packet timestamp source (if available).
Some clients, even if have `Expect: 100-continue` in request, start
sending data without waiting approval from the server. More over, there
is cases when response “100 Continue” is received, client changes TCP
seq, and following requests start having valid sequences, based on
response one.
This change should address this bugs.
As small addition extended list of http methods, to fully support
webdav.
