In some low traffic cases you can have cases when time between request minutes.
Additionally increasing speed can be not an option.
Now you can "skip" this pauses, by seetting max wait time
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Input file now pre-reads N requests, sort them by timestamp and emit on demand.
You can control read depth using --input-file-read-depth which is 100 by default.
It makes implementaiton faster, and it fix various issues when due to concurrenccy, or another issues requests gets addeed out of order.
Windows having issues with generating timestamps, so adding application level timestamp generation
Made small refactoring to move "accurate-enough" time to own package.
In some rare cases, requests can be recorded out of order.
It was causing replay from file significanty slodown when such issue met.
Now it replays them but without adding sleep.
When you use S3 and write from multiple machines, you need to ensure that file will have unique name for each machine.
This PR adds special %i function which you can use in file templates, which will add some random process uniq string to the file.
Added `--input-raw-allow-incomplete` if you really need it.
Fixed Bug when outpuut binary response not tracked
Additionally fixed bug which prevents Gor from exiting.
Do not use sync.Pool, use channel based impementation instead. Sync pool does not work for long lived objects, and trigger frequent GC.
Because of MUCH more efficient memory utilization, GOGC now set to 500, which significantly improve CPU as wel.
The cause was not ZeroCopy but wrong SetTimeout (block cause a lot of overhead).
Packet processing previously used channels, but now, with ring buffer, using select was causing issue. Adding `default` clause fixed the issue.
It is a high performant alternative to libpcap engine.
Only Linux supported.
Performance gain can be up to 50% depending on traffic type.
Can be enabled using:
`--input-raw-engine af_packet`
Co-authored-by: Leonid Bugaev <leonsbox@gmail.com>
Added a way to capture multiple ports at the same time, with a single listener.
Ports should be separated by comma like this: --input-raw :80,8080,3000
Issues is that Go built-in net.Interfaces function in newer Windows versions return wrong interface names, which libpcap can't consume.
Now we use pcap.FindDevices instead of net.Interfaces.
See this Article for deep understanding of the issue https://haydz.github.io/2020/07/06/Go-Windows-NIC.html
Additionally, found a bug causing big memory allocations, for large requests, when we perform check if messages finished or not.
Because of this bug chunked body encoding check was not working properly.
Was not caught in tests, because test was working on packet array level, and this issue happens when dealing with TCP message object.
Additionally, added a small fix for windows Makefile task, it now generates proper file name.
For a long time there were no official binaries for the windows platform.
One of the reasons is the complexities of the build toolchain. Not only CGO is required, but also installing the needed libraries and header files, not talking about mingw and etc.
2 weeks ago [golang-crossbuild](https://github.com/elastic/golang-crossbuild) project added native support for Libpcap based applications.
Windows support is based on the WinPcap which is a bit (a lot) outdated, BUT, since we depend only on its interface, it is still possible to use projects like npcap https://nmap.org/npcap/#download.
Npcap needs be installed with WinPcap compatibe mode (checkbox during installation)
It is also possibe install it in silent mode like this: `npcap-0.86.exe /S /winpcap_mode=yes`
After PR merge, will be updated related documentation.
Additionally fix dependency on "unix" package (apparently it can be totally replaced using universal syscall package)
For a long time there were no official binaries for the windows platform.
One of the reasons is the complexities of the build toolchain. Not only CGO is required, but also installing the needed libraries and header files, not talking about mingw and etc.
2 weeks ago [golang-crossbuild](https://github.com/elastic/golang-crossbuild) project added native support for Libpcap based applications.
Windows support is based on the WinPcap which is a bit (a lot) outdated, BUT, since we depend only on its interface, it is still possible to use projects like npcap https://nmap.org/npcap/#download.
Npcap needs be installed with WinPcap compatibe mode (checkbox during installation)
It is also possibe install it in silent mode like this: `npcap-0.86.exe /S /winpcap_mode=yes`
After PR merge, will be updated related documentation.
Additionally fix dependency on "unix" package (apparently it can be totally replaced using universal syscall package)
### Reducing CPU context switching and number of goroutines.
Packet capture and packet processing now use only two goroutines which helps to minimize CPU context switches. Spawning too many goroutines is harmful here.
### Optimized packet capture - allocated memory only when required, and only for data which is used
Using ZeroCopy methods from libpcap library to avoid unnecessary allocations. Now memory gets allocated ONLY for the valid packets, and only for the packets which have the data. E.g. no SYN/FIN packets are used now. Additionally we now use `sync.Pool` for re-using packet objects, which helps to re-use already allocated memory.
### Simplification and optimization of request/response detection
There is no SYN/FIN packets anymore etc. Now only packet payload is used to detect start and end of the packet. More over payload detection now does not require generating a total “message” buffer, and works with individual packet payloads.
Message payloads now concatenated from packets only in the end when message is dispatched. Also, before checking if message is complete, added additional check if all received packets in the valid order, e.g. if their SEQ is valid, and no packets are missing.
Reworked chunked encoding validation, and now it does not need expensive operation of re-calculating all the chunks. Now it “trust” that client gives valid chunk body, check if packets are in the right order (e.g. SEQ match), and checks if message ends with the right suffix. All is done with 0 allocations.
Parsing all Headers using `proto.GetHeaders` was proved to be very slow. Now we only parse the headers we need(and do it only once).
Packets gets matched together using ACK, which on high RPS removed chances of duplicating IDs. Additionally, even if packets are received out of order, now it will properly sort them, before dispatching the message.
### Changes in ID generation algorithm
Message ID generation and relations between request and response IDs is fully rewritten. Responses now do not have to lookup for request data in order to get the same ID. ID no rely on the fact that SEQ of the first packet of the response should be the same as ACK of the request. If previously Message ID contained random values, like current timestamp, now it has a consistent algorithm which is based on TCP stream id (SrcPort + DstPort + SrcIP/DstIP) and current ACK/SEQ number (to distinguish multiple messages within the same stream).
### BPF filter optimizations
When tracking response it now uses a more accurate BPF rule to filter only needed traffic.
### Misc
The packet code is now fully moved to tcp/Packet, so packet processing done only once in one place.
TCP output now has a 5 second timeout, and has a proper Close method.
Fully switching to go modules and removing vendoring.
Leonid Bugaev