1314 Commits
Author SHA1 Message Date
Leonid BugaevandGitHub fe25ddd0fb Merge pull request #982 from DimaGolomozy/testing-fix-work
Testing fix work
2021-08-16 20:07:31 +03:00
dimagolomozy 56ed4760f0 capture test 2021-08-16 19:23:54 +03:00
Leonid BugaevandGitHub ca90d796fb Merge pull request #995 from DimaGolomozy/fix-Src-Dst-Add-in-message
when using the `RealIpHeader` flag, it tries to append the real ip from the msg as a header
but the `SrcAddr` was never assigned with the IP, so we get empty string all the time
![image](https://user-images.githubusercontent.com/9161830/129594595-96c0cd53-a28f-4b72-9f6d-c782e23af5dc.png)
2021-08-16 19:11:40 +03:00
dimagolomozy f4895e00d5 no Src and Dst addr in message 2021-08-16 19:05:57 +03:00
Leonid BugaevandGitHub 4d82e2f652 Merge pull request #994 from DimaGolomozy/fix-promisc-mode
if its promiscuous mode, no need to filter on host as we want to capture traffic not related to this host
2021-08-16 18:56:22 +03:00
dimagolomozy b50ea90cdb if its promiscuous mode, no need to filter on host as we want to capture traffic not related to host 2021-08-16 18:21:32 +03:00
Leonid BugaevandGitHub 42d89903b6 Merge pull request #992 from buger/fix/989-duplicate
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e

Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.

Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.

Fix #989
v1.3.1
2021-08-12 22:50:08 +03:00
Leonid Bugaev 9bdf27226d Fix dupicated requests on linux
Issue was introduced while fixing windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e

Added exception for Windows, which by default allows interfaces without IPs.
Interface name check moved higher, so if interface namee or IP match, rest of check will be ignored.

Additionally windows npcap loopback mechanism can now be picked by specifying 127.0.0.1 or loopback IP.

Fix #989
2021-08-12 22:45:24 +03:00
Leonid Bugaev 11d61dcb4d Make packet proccessing multi threaded
If packet processign is not fast enough, packet capture may miss packets
Now it use 10 goroutines, which distribute work based on the ephemeral port number
2021-08-06 21:24:18 +03:00
dimagolomozy 3211d251b9 fix? 2021-08-04 14:09:40 +03:00
dimagolomozy 6f48d23953 ignore bad tests 2021-08-04 13:51:51 +03:00
dimagolomozy bf9d233ab0 fix echo.sh test 2021-08-04 13:48:36 +03:00
Leonid Bugaev c9274ac92a Fix windows loopback
Windows loopback do not have addresses
v1.3.0 v1.3.0_RC11
2021-08-03 17:42:13 +03:00
dimagolomozy 9e2754f673 update name only when writing 2021-08-03 17:07:49 +03:00
Leonid BugaevandGitHub 024e1261a5 Merge pull request #981 from DimaGolomozy/fix-sticky-buffer-index
fix buffer index number for sticky tcp output connection
2021-08-03 16:05:11 +03:00
dimagolomozy f11d59ba19 fix sleep time 2021-08-03 15:21:46 +03:00
dimagolomozy f115627991 change debug level 2021-08-03 15:17:41 +03:00
dimagolomozy 524b9dfa61 fix buffer index number for sticky tcp output connection 2021-08-03 14:28:39 +03:00
Leonid BugaevandGitHub 0f2f93a60f Merge pull request #974 from lins05/fix-file-output-size-limit
This should fix #973 . Tested with the following command:

```bash
gor --input-raw 127.0.0.1:8080 \
      --output-file '%Y-%m-%d.log' \
       --output-file-queue-limit 0 \
      --output-file-size-limit 1mb
```
2021-07-27 20:28:32 +03:00
Leonid Bugaev 214edb45f6 Optimise packet capture thread
Ensure that thread which capture packets as fast as possible.
Packet parsing logic moved to different threads.
Additionally using os.LockOsThread to reduce CPU context switching
2021-07-26 21:01:59 +03:00
Leonid Bugaev 79ff8823a3 Add packet capture debug stats
Also remove potentially sensitive memstat and cmdargs from /debug/vars endopint
2021-07-26 15:38:39 +03:00
Shuai Lin debc3b4384 Applied gofmt 2021-07-25 23:39:21 +08:00
Shuai Lin f77fd8af57 Fixed --output-file-size-limit 2021-07-25 23:39:07 +08:00
Leonid Bugaev c01be50d72 Fix issue with multi-packet messages and fix tests 2021-07-19 20:36:12 +03:00
Leonid Bugaev df2b4be898 Fix detection of packet detection for complex bpf fiters
Right now it depends on the defined ports, but you can have more compex bpf, so wee may stil need ti check it with Start function
2021-07-16 22:18:28 +03:00
Leonid Bugaev 0902a10c2d Rollback to use non ZeroCopy methods, and remove poling 2021-07-13 23:37:39 +03:00
Leonid Bugaev 710cbe10e3 Simplify packet pooling and improve copySlice
Pooling now use just sync.Pool, copySlice using trick from gopacket library
2021-07-12 23:24:22 +03:00
Leonid BugaevandGitHub b79483b37f Merge pull request #962 from wangfeng22/patch-1
Fix a spelling error
2021-07-12 20:17:21 +03:00
旺汪andGitHub ec1dcccc31 Update settings.go
Fix a spelling error
2021-07-12 11:31:21 +08:00
Leonid BugaevandGitHub da04e88a56 Merge pull request #961 from buger/feature/file-improvements
# Reading in advance
Input file now pre-reads N requests (and keeps this buffer on the same length), sort them by timestamp and emit on demand.
You can control read depth using `--input-file-read-depth` which is 100 by default.

It makes implementation faster, and it fixes various issues when due to concurrency, or another issues requests gets added out of order.

Setting too big depth will mean that memory consumption will be bigger, since it will need store request in memory.

# Max wait
In some low traffic cases, you can have cases when time between request minutes.
Increasing speed can be not an option.
Now you can "skip" this pauses, by setting max wait time. Example: `--input-raw-max-wait 1s`.

# Dry-run mode
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
Example report:
```
Records found: 258
Files processed: 1
Bytes processed: 208343
Max wait: 1h26m3.193159s
Min wait: 15µs
First wait: 1.174ms
It will take `1h41m44.614806s` to replay at current speed.
```
Applying options like input-raw-max-wait, setting speed, or read depth affecting dry-run mode as well.

# Misc
Fix issue with re-using buffers for messages. On concurrency, buffers gets re-used, and you get corrupted data.
2021-07-08 23:32:36 +03:00
Leonid Bugaev 19ad90aa4b Added time --input-raw-max-wait
In some low traffic cases you can have cases when time between request minutes.
Additionally increasing speed can be not an option.
Now you can "skip" this pauses, by seetting max wait time
2021-07-08 23:23:02 +03:00
Leonid Bugaev 8e76559b49 Add --input-file-dry-run option
Now you can get information about file content without performing the actual replay.
For example, it can tell you how many requests in your files, and how long it will take to replay them.
2021-07-08 23:09:50 +03:00
Leonid Bugaev 61b377d123 Do not use buffer pool for messages
This buffers are get re-used and data gets corrupted
2021-07-08 23:00:12 +03:00
Leonid Bugaev 625ed54f1e Allow input file read ahead
Input file now pre-reads N requests, sort them by timestamp and emit on demand.
You can control read depth using --input-file-read-depth which is 100 by default.

It makes implementaiton faster, and it fix various issues when due to concurrenccy, or another issues requests gets addeed out of order.
2021-07-08 10:30:22 +03:00
Leonid Bugaev 889c1e6d41 Fix windows interface matching
In windows pcap.Name differ from net.Interface Name, so instead we have to match by interface addrs
2021-07-07 21:11:11 +03:00
Leonid Bugaev 8edb74e572 Add support for "go" timestamp source
Windows having issues with generating timestamps, so adding application level timestamp generation
Made small refactoring to move "accurate-enough" time to own package.
2021-07-07 20:56:22 +03:00
Leonid Bugaev 59b12abbe1 Fix linux compilation issue 2021-07-07 19:33:49 +03:00
Leonid Bugaev 5d718e92ea Add exvar to expose debug variables 2021-07-07 12:30:13 +03:00
Leonid Bugaev 148846be93 GC optimisations 2021-07-07 03:04:28 +03:00
Leonid Bugaev 0bb72e0f60 Add buffer pooling for message data 2021-07-07 02:58:00 +03:00
Leonid Bugaev af19f0feb8 Remove GOGC trick 2021-07-07 02:24:48 +03:00
Leonid Bugaev 724c0890c9 Improve memory allocations
A lot of improvements, and own GC for packet pool
2021-07-07 02:08:23 +03:00
Leonid Bugaev a727ade006 Fix BPF filter when listen on all interfaces
It was caching bpf filter of first interface
2021-07-06 19:13:48 +03:00
Leonid Bugaev 92c63de753 Fix replay of requests with timestampp in "past"
In some rare cases, requests can be recorded out of order.
It was causing replay from file significanty slodown when such issue met.
Now it replays them but without adding sleep.
2021-07-01 23:27:34 +03:00
Leonid Bugaev b6cc6a3606 Add a way to add process ID to file names
When you use S3 and write from multiple machines, you need to ensure that file will have unique name for each machine.
This PR adds special %i function which you can use in file templates, which will add some random process uniq string to the file.
2021-07-01 21:22:03 +03:00
Leonid Bugaev 2b993eda49 Fix --input-raw-bpf-filter
It was just ignored

Should fix #955
2021-07-01 09:17:41 +03:00
Leonid Bugaev de8a74fdca Fix compilation error 2021-07-01 09:17:20 +03:00
Leonid Bugaev f701dace50 Fix MacOS builds v1.3.0_RC6 2021-06-29 23:10:39 +03:00
Leonid Bugaev ad43405793 Fix MacOS compilation 2021-06-29 21:11:31 +03:00
Leonid BugaevandGitHub 9468978353 Skip not complete HTTP messages (#953)
Added `--input-raw-allow-incomplete` if you really need it.

Fixed Bug when outpuut binary response not tracked
Additionally fixed bug which prevents Gor from exiting.
2021-06-28 23:01:58 +03:00