diff --git a/docs/Capturing-and-replaying-traffic.md b/docs/Capturing-and-replaying-traffic.md new file mode 100644 index 0000000..0e37fd9 --- /dev/null +++ b/docs/Capturing-and-replaying-traffic.md @@ -0,0 +1,49 @@ +Think about Gor more like a network analyzer or tcpdump on steroids, it is not a proxy and does not affect your app anyhow. You specify application port, and it will capture and replay incoming data. + +Simplest setup will be: +```bash +# Run on servers where you want to catch traffic. You can run it on every `web` machine. +sudo gor --input-raw :80 --output-http http://staging.com +``` +It will record and replay traffic from the same machine. However, it is possible to use [[Aggregator-forwarder setup]], when Gor on your web machines forward traffic to Gor aggregator instance running on the separate server. + +> You may notice that it require `sudo`: to analyze network Gor need permissions which available only to root users. However, it is possible to configure Gor [beign run for non-root users](Running as a non-root user). + + +### Forwarding to multiple addresses + +You can forward traffic to multiple endpoints. +``` +gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com" +``` + +### Splitting traffic +By default, it will send same traffic to all outputs, but you have options to equally split it (round-robin) using `--split-output` option. + +``` +gor --input-raw :80 --output-http "http://staging.com" --output-http "http://dev.com" --split-output true +``` + +### Tracking responses +By default `input-raw` does not intercept responses, only requests. You can turn response tracking using `--input-raw-track-response` option. When enable you will be able to access response information in middleware and `output-file`. + + +### Traffic interception engine +By default, Gor will use `libpcap` for intercepting traffic, it should work in most cases. If you have any troubles with it, you may try alternative engine: `raw_socket`. + +``` +sudo gor --input-raw :80 --input-raw-engine "raw_socket" --output-http "http://staging.com" +``` + +You can read more about [[Replaying HTTP traffic]]. + + +### Tracking original IP addresses +You can use `--input-raw-realip-header` option to specify header name: If not blank, injects header with given name and real IP value to the request payload. Usually, this header should be named: `X-Real-IP`, but you can specify any name. + +`gor --input-raw :80 --input-raw-realip-header "X-Real-IP" ...` + + +*** + +Also you may want to know about [[Rate limiting]], [[Request rewriting]] and [[Request filtering]] \ No newline at end of file diff --git a/docs/Commercial-FAQ.md b/docs/Commercial-FAQ.md new file mode 100644 index 0000000..9070f0a --- /dev/null +++ b/docs/Commercial-FAQ.md @@ -0,0 +1,58 @@ +### What are GoReplay PRO and GoReplay Enterprise? + +[GoReplay PRO](https://goreplay.org/pro.html) and GoReplay Enterprise are extensions to GoReplay which add more functionality and provide additional support options for customers. + +### Is there a trial version? + +There's no free trial but we do offer a 14 day period with full refund if it does not work for you. + +### What is the license? + +See [COMM-LICENSE](https://github.com/buger/gor/blob/master/COMM-LICENSE) in the root of the GoReplay repo. + +### How does PRO licensing work? + +Every organization running GoReplay Pro on its own servers must have a license. There's no limit to the amount of servers or environments used by that organization. + +### How does Enterprise licensing work? + +Every organization running Gor Enterprise on its own servers must have a license. There's **no limit** to the amount of servers or environments used by that organization. + +### What happens if my subscription lapses? + +You must have an active subscription to run GoReplay Pro or Enterprise. After a one week grace period, you'll lose access to binaries and priority support. You won't get any more updates or bug fixes. + +### How do I buy GoReplay Enterprise? + +Send email to [support@gortool.com](mailto:support@gortool.com) with your info. A PDF quote will be emailed to you with the price. Reply to that email with your purchase order or just "Sounds good" and we will send an invoice which can be paid with a credit card, ACH bank transfer or a paper check. + +### Can I upgrade from GoReplay Pro? + +Yes! Current subscribers can upgrade by [requesting a quote](mailto:support@gortool.com). **Please note that you are an existing Pro subscriber.** We will add a one-time discount on your first invoice to reflect any remaining Pro subscription credit. If you purchased GoReplay Pro 6 months ago for $950, you'll get a $475 discount. + +### Can I distribute GoReplay PRO or Enterprise to my customers? + +This is a common requirement for "on-site installs" or "appliances" sold to large corporations. + +The standard license is appropriate for SaaS usage as it does not allow distribution. GoReplay PRO and Enterprise have an Appliance license option which **does** allow you to distribute them. The Appliance license is $9,500/yr for Pro and $19,500/yr for Enterprise. It allows you to distribute the Pro or Enterprise binaries as part of your application and each of your customers to run GoReplay Pro or Enterprise. Email [support@gortool.com](mailto:support@gortool.com) to purchase. + +### Can you transfer a license? + +Licenses are **not** transferable to another company. We will transfer the license from a user-specific email to a group email address (e.g. john_smith@example.com -> tech@example.com) but only for **the same domain**. It is strongly recommended that you buy the license using a group email address so the license is not attached to any one employee's email address. + +### What does the license require me to do? + +Your purchase gets you a unique access URL for downloading the Pro and/or Enterprise binaries. The license agreement requires you to keep this access URL private. If we find your access URL is ever publicized: + +1. We'll send you a warning email with details. You need to remove the content and send a new email address so we can generate a new access URL for you. The old access URL will stop working immediately so you'll need to update your apps. +2. If your access URL is publicized a second time, we reserve the right to permanently remove access. + +### Can I get a refund? + +Yes, up to two weeks after purchase. Let us know the reason and maybe we can help but either way it's not a problem. Email [support@gortool.com](mailto:support@gortool.com). + +### How do I update my credit card info? + +If you purchased GoReplay Enterprise, there's nothing to do. Each annual invoice is paid separately. + +If you purchased GoReplay PRO, log into [Gumroad](https://gumroad.com) with your email address, click the Billing tab and enter your new card. I can't provide support for the Gumroad website and don't have the ability to edit customer info - if you can't log in or change your credit card, you can always let your current subscription expire and purchase a new subscription. \ No newline at end of file diff --git a/docs/Commercial-Support.md b/docs/Commercial-Support.md new file mode 100644 index 0000000..105fb58 --- /dev/null +++ b/docs/Commercial-Support.md @@ -0,0 +1,11 @@ +Gor offers only community support. Gor Pro and Enterprise offer priority support via email. + +## Priority Support + +Covers 1 incident per quarter, with a max response time of 2 working days. Scope is limited to Gor and Gor Pro and Enterprise features and APIs, not the application or infrastructure. For support, email **support** AT **gortool.com**. Please email using the same domain as the original license email or explain your connection to the licensed company. + +More aggressive support contracts (phone, quicker response time) are available separately, email with your needs. + +## Onboarding + +Enterprise customers may request a one hour video chat session with @buger to discuss their application(s), requirements and how best to leverage the various Gor features. Contact support to set up your session. \ No newline at end of file diff --git a/docs/Commercial-collaboration.md b/docs/Commercial-collaboration.md new file mode 100644 index 0000000..977cbcf --- /dev/null +++ b/docs/Commercial-collaboration.md @@ -0,0 +1,19 @@ +Collaboration is difficult with commercial closed source but I do want to keep as much of the OSS ethos as possible available to customers who want to fix it themselves. + +## Legal + +In order to unambiguously own and sell Gor commercial products, I must have the copyright associated with the entire codebase. Any code you create which is merged must be owned by me. That's not me trying to be a jerk, that's just the way it works. + +## Application + +If you wish access to the product repository so you can send a PR, just open a new Gor issue and include the following info: + +1. the email address that bought the license, a max of one collaborator per license +1. the following statement "I assign all rights, including copyright, to any future Gor work by myself to Leonid Bugaev" + +You should be granted access to the private repo soon after. + +## Notes + +1. You should **never** work on the master branch. Only I may merge changes. +1. I may revoke access for any reason at any time. Access is not guaranteed with purchase. \ No newline at end of file diff --git a/docs/Compilation.md b/docs/Compilation.md new file mode 100644 index 0000000..8eb38fd --- /dev/null +++ b/docs/Compilation.md @@ -0,0 +1,30 @@ +We provide pre-compiled binaries for Mac and Linux, but you are free to compile Gor by yourself. + +Gor is written using Go, so first you need to download it from here https://golang.org/, use the latest stable version. + +The only Gor dependency is [libpcap](https://github.com/the-tcpdump-group/libpcap), which is the interface to various kernel packet capture mechanisms, and https://github.com/google/gopacket, which is a Go wrapper around libpcap. Latest libpcap version can be obtained at http://www.tcpdump.org/release/. Libpcap itself depend on `flex` and `bison` packages, many operating systems already have them installed. + +```bash +# Fetch libpcap dependencies. Depending on your OS, instead of `apt` you will use `yum` or `rpm`, or `brew` on Mac. +sudo apt-get install flex bison -y + +# Download latest stable release, compile and install it +wget http://www.tcpdump.org/release/libpcap-1.7.4.tar.gz && tar xzf libpcap-1.7.4.tar.gz +cd libpcap-1.7.4 +./configure && make install + + +# Lets fetch Gor source code +mkdir $HOME/gocode +# See more information about GOPATH https://github.com/golang/go/wiki/GOPATH +export GOPATH=$HOME/gocode +# Fetch code from the Github +go get github.com/buger/gor + +# Compile from source +cd $HOME/gocode/src/github.com/buger/gor +go build LDFLAGS = -ldflags "-extldflags \"-static\"" +``` + +After you finished, you should see `gor` binary in current directory. + diff --git a/docs/Development-Setup.md b/docs/Development-Setup.md new file mode 100644 index 0000000..8e3d954 --- /dev/null +++ b/docs/Development-Setup.md @@ -0,0 +1,39 @@ +## STEP 1: Install Docker +For local development we recommend to use Docker. + +If you don’t have it you can read how to install it here: +https://docs.docker.com/engine/getstarted/step_one/#step-3-verify-your-installation + +## STEP 2: Download repository + +`git clone git@github.com:buger/goreplay.git` + + +## STEP 3: Setup container + +``` +cd ./goreplay +make build + +``` + +## Testing +To run tests execute next command: + +``` +make test +``` + +You can copy the command that is produced and modify it. For example, if you need to run one test copy the command and add `-run TestName`, e.g.: + +``` +docker run -v `pwd`:/go/src/github.com/buger/gor/ -p 0.0.0.0:8000:8000 -t -i gor:go go test ./. -run TestEmitterFiltered -timeout 60s -ldflags "-X main.VERSION=DEV-1482398347 -extldflags \"-static\"" -v +``` + + +## Building +To get a binary file run + +``` +make release-bin +``` diff --git a/docs/Distributed-configuration.md b/docs/Distributed-configuration.md new file mode 100644 index 0000000..2795bf8 --- /dev/null +++ b/docs/Distributed-configuration.md @@ -0,0 +1,26 @@ +Sometimes it makes sense to use separate Gor instance for replaying traffic and performing things like load testing, so your production machines do not spend precious resources. It is possible to configure Gor on your web machines forward traffic to Gor aggregator instance running on the separate server. + +```bash +# Run on servers where you want to catch traffic. You can run it on each `web` machine. +sudo gor --input-raw :80 --output-tcp replay.local:28020 + +# Replay server (replay.local). +gor --input-tcp replay.local:28020 --output-http http://staging.com +``` + +If you have multiple replay machines you can split traffic among them using `--split-output` option: it will equally split all incoming traffic to all outputs using round robin algorithm. +``` +gor --input-raw :80 --split-output --output-tcp replay1.local:28020 --output-tcp replay2.local:28020 +``` + +[GoReplay PRO](https://goreplay.com/pro.html) support accurate recording and replaying of tcp sessions, and when `--recognize-tcp-sessions` option is passed, instead of round-robin it will use a smarter algorithm which ensures that same sessions will be sent to the same replay instance. + + +In case if you are planning a large load testing, you may consider use separate master instance which will control Gor slaves which actually replay traffic. For example: +``` +# This command will read multiple log files, replay them on 10x speed and loop them if needed for 30 seconds, and will distributed traffic (tcp session aware) among multiple workers +gor --input-file logs_from_multiple_machines.*|1000% --input-file-loop --exit-after 30s --recognize-tcp-sessions --split-output --output-tcp worker1.local --output-tcp worker2.local:27017 --output-tcp worker3.local:27017 ... --output-tcp workerN.local:27017 + +# worker +gor --input-tcp :27017 --ouput-http load_test.target +``` diff --git a/docs/Exporting-to-ElasticSearch.md b/docs/Exporting-to-ElasticSearch.md new file mode 100644 index 0000000..228ccdb --- /dev/null +++ b/docs/Exporting-to-ElasticSearch.md @@ -0,0 +1,38 @@ +Gor can export requests and replayed response data to ElasticSearch: + +``` +./gor --input-raw :8000 --output-http http://staging.com --output-http-elasticsearch localhost:9200/gor +``` + +You don't have to create the index upfront. That will be done for you automatically. + +### Format + +Following structure represents ES format: + +``` +type ESRequestResponse struct { + ReqURL string `json:"Req_URL"` + ReqMethod string `json:"Req_Method"` + ReqUserAgent string `json:"Req_User-Agent"` + ReqAcceptLanguage string `json:"Req_Accept-Language,omitempty"` + ReqAccept string `json:"Req_Accept,omitempty"` + ReqAcceptEncoding string `json:"Req_Accept-Encoding,omitempty"` + ReqIfModifiedSince string `json:"Req_If-Modified-Since,omitempty"` + ReqConnection string `json:"Req_Connection,omitempty"` + ReqCookies string `json:"Req_Cookies,omitempty"` + RespStatus string `json:"Resp_Status"` + RespStatusCode string `json:"Resp_Status-Code"` + RespProto string `json:"Resp_Proto,omitempty"` + RespContentLength string `json:"Resp_Content-Length,omitempty"` + RespContentType string `json:"Resp_Content-Type,omitempty"` + RespTransferEncoding string `json:"Resp_Transfer-Encoding,omitempty"` + RespContentEncoding string `json:"Resp_Content-Encoding,omitempty"` + RespExpires string `json:"Resp_Expires,omitempty"` + RespCacheControl string `json:"Resp_Cache-Control,omitempty"` + RespVary string `json:"Resp_Vary,omitempty"` + RespSetCookie string `json:"Resp_Set-Cookie,omitempty"` + Rtt int64 `json:"RTT"` + Timestamp time.Time +} +``` \ No newline at end of file diff --git a/docs/FAQ.md b/docs/FAQ.md new file mode 100644 index 0000000..a30c1c8 --- /dev/null +++ b/docs/FAQ.md @@ -0,0 +1,32 @@ +### What OS are supported? +Gor will run everywhere where [libpcap](http://www.tcpdump.org/) works, and it works on most of the platforms. However, currently, we test it on Linux and Mac. See more about [[Compilation]]. + +### Why does the `--input-raw` requires sudo or root access? +Listener works by sniffing traffic from a given port. It's accessible +only by using sudo or root access. But it is possible to [[Running as non root user]]. + +### How do you deal with user session to replay the traffic correctly? +You can rewrite session related headers/params to match your staging environment. If you require custom logic (e.g random token based auth) follow this discussion: https://github.com/buger/gor/issues/154 + +### Can I use Gor to intercept SSL traffic? +Basic idea is that SSL was made to protect itself from traffic interception. There 2 options: +1. Move SSL handling to proxy like Nginx or Amazon ELB. And allow Gor to listen on upstreams. +2. Use `--input-http` so you can duplicate request payload directly from your app to Gor, but it will require your app modifications. + +More can be find here: https://github.com/buger/gor/issues/85 + +### Is there a limit for size of HTTP request when using output-http? +Due to the fact that Gor can't guarantee interception of all packets, for large payloads > 200kb there is chance of missing some packets and corrupting body. Treat it as a feature and chance to test broken bodies handling :) +The only way to guarantee delivery is using `--input-http`, but you will miss some features. + +### I'm getting 'too many open files' error +Typical Linux shell has a small open files soft limit at 1024. You can easily raise that when you do this before starting your gor replay process: + + ulimit -n 64000 + +More about ulimit: http://www.thecodingmachine.com/solving-the-too-many-open-files-exception-in-red5-or-any-other-application/ + +### The CPU average across my load-balanced targets is higher than the source +If you are replaying traffic from multiple listeners to a load-balanced target and you use sticky sessions, you may observe that the target servers have a higher CPU load than the listener servers. This may be because the sticky session cookie of the original load balancer is not honored by the target load balancer thus resulting in requests that would normally hit the same target server hitting different servers on the backend thus reducing some caching benefits gained via the load balancing. Try running just one listener against one replay target and see if the CPU utilization comparison is more accurate. + +Also see [[Troubleshooting]]. \ No newline at end of file diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md new file mode 100644 index 0000000..6d16f63 --- /dev/null +++ b/docs/Getting-Started.md @@ -0,0 +1,44 @@ +### Dependencies +To start working with Gor, you need to have a web server running on your machine, and a terminal to run commands. If you are just poking around, you can quickly start the server by calling `gor file-server :8000`, this will start a simple file server of the current directory on port `8000`. + +### Installing Gor +Download the latest Gor binary from https://github.com/buger/gor/releases (we provide precompiled binaries for Windows, Linux x64 and Mac OS), or you can compile by yourself [[Compilation]]. + +Once the archive is downloaded and uncompressed, you can run Gor from the current directory, or you may want to copy binary to your PATH (for Linux and Mac OS it can be `/usr/local/bin`). + +### Capturing web traffic +Now run this command in terminal: `sudo ./gor --input-raw :8000 --output-stdout` + +This command says to listen for all network activity happening on port 8000 and log it to stdout. +If you are familiar with `tcpdump`, we are going to implement similar functionality. + +> You may notice that it uses `sudo` and asks for the password: to analyze network, Gor needs permissions which are available only to super users. +> However, it is possible to configure Gor [being run for non-root users](Running-as-non-root-user). + + +Make a few requests by opening `http://localhost:8000` in your browser, or just by calling curl in terminal `curl http://localhost:8000`. You should see that `gor` outputs all the HTTP requests and responses right to the terminal window where it is running. + + +**Gor is not a proxy:** you do not need to put 3-rd party tool to your critical path. Instead Gor just silently analyzes the traffic of your application and does not affect it anyhow. + +### Replaying + +Now it's time to replay your original traffic to another environment. Let's start the same file web server but on a different port: `gor file-server :8001`. + +Instead of `--output-stdout` we will use `--output-http` and provide URL of second server: `sudo ./gor --input-raw :8000 --output-http="http://localhost:8001"` + +Make few requests to first server. You should see them replicated to the second one, voila! + +### Saving requests to file and replaying them later +Sometimes it's not possible to replay requests in real time; Gor allows you to save requests to the file and replay them later. + +First use `--output-file` to save them: `sudo ./gor --input-raw :8000 --output-file=requests.gor`. This will create new file and continuously write all captured requests to it. + +Let's re-run Gor, but now to replay requests from file: `./gor --input-file requests.gor --output-http="http://localhost:8001"`. You should see all the recorded requests coming to the second server, and they will be replayed in the same order and with exactly same timing as they were recorded. + +Next: [[The Basics]] + +### Watch an overview: + + +[![Foo](http://img.youtube.com/vi/CxuKZcMKaW4/0.jpg)](https://www.youtube.com/watch?v=CxuKZcMKaW4) \ No newline at end of file diff --git a/docs/Home.md b/docs/Home.md new file mode 100644 index 0000000..88a7bc9 --- /dev/null +++ b/docs/Home.md @@ -0,0 +1,29 @@ +Gor is an open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data. It can be used to increase confidence in code deployments, configuration changes and infrastructure changes. + +Read for more info: + +* [[Getting Started]] +* [[The Basics]] +* [[Capturing and replaying traffic]] +* [[Replaying HTTP traffic]] +* [[[PRO] Replaying Binary protocols]] +* [[[PRO] Recording and replaying keep alive TCP sessions]] +* [[Saving and Replaying from file]] +* [Performance testing](https://github.com/buger/gor/wiki/Saving-and-Replaying-from-file#performance-testing) +* [[Rate limiting]] +* [[Request filtering]] +* [[Request rewriting]] +* [[Middleware]] +* [[Distributed configuration]] +* [[Exporting to ElasticSearch]] +* [[FAQ]] +* [[Troubleshooting]] + +## Commercial Aspects + +* [[Commercial Support]] +* [[Commercial FAQ]] +* [[Commercial collaboration]] + + +Next: [Getting Started](Getting-Started) \ No newline at end of file diff --git a/docs/Middleware.md b/docs/Middleware.md new file mode 100644 index 0000000..7c98cbe --- /dev/null +++ b/docs/Middleware.md @@ -0,0 +1,74 @@ +#### Overview +Middleware is a program that accepts request and response payload at STDIN and emits modified requests at STDOUT. You can implement any custom logic like stripping private data, advanced rewriting, support for oAuth and etc. Check examples [included into our repo](https://github.com/buger/gor/tree/master/examples/middleware). + + +``` + Original request +--------------+ ++-------------+----------STDIN---------->+ | +| Gor input | | Middleware | ++-------------+----------STDIN---------->+ | + Original response (1) +------+---+---+ + | ^ ++-------------+ Modified request v | +| Gor output +<---------STDOUT-----------------+ | ++-----+-------+ | + | | + | Replayed response | + +------------------STDIN----------------->----+ +``` + +(1): Original responses will only be sent to the middleware if the `--input-raw-track-response` option is specified. + +Middleware can be written in any language, see `examples/middleware` folder for examples. +Middleware program should accept the fact that all communication with Gor is asynchronous, there is no guarantee that original request and response messages will come one after each other. Your app should take care of the state if logic depends on original or replayed response, see `examples/middleware/token_modifier.go` as example. + +Simple bash echo middleware (returns same request) will look like this: +```bash +while read line; do + echo $line +end +``` + +Middleware can be enabled using `--middleware` option, by specifying path to executable file: +``` +gor --input-raw :80 --middleware "/opt/middleware_executable" --output-http "http://staging.server" +``` + +#### Communication protocol +All messages should be hex encoded, new line character specifieds the end of the message, eg. new message per line. + +Decoded payload consist of 2 parts: header and HTTP payload, separated by new line character. + +Example request payload: + +``` +1 932079936fa4306fc308d67588178d17d823647c 1439818823587396305 +GET /a HTTP/1.1 +Host: 127.0.0.1 + +``` + +Example response payload (note: you will only receive this if you specify `--input-raw-track-response`) + +``` +2 8e091765ae902fef8a2b7d9dd960e9d52222bd8c 1439818823587996305 2782013 +HTTP/1.1 200 OK +Date: Mon, 17 Aug 2015 13:40:23 GMT +Content-Length: 0 +Content-Type: text/plain; charset=utf-8 + +``` + +Header contains request meta information separated by spaces. First value is payload type, possible values: `1` - request, `2` - original response, `3` - replayed response. +Next goes request id: unique among all requests (sha1 of time and Ack), but remain same for original and replayed response, so you can create associations between request and responses. The third argument is the time when request/response was initiated/received. Forth argument is populated only for responses and means latency. + +HTTP payload is unmodified HTTP requests/responses intercepted from network. You can read more about request format [here](http://www.jmarshall.com/easy/http/), [here](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) and [here](http://www.w3.org/Protocols/rfc2616/rfc2616.html). You can operate with payload as you want, add headers, change path, and etc. Basically you just editing a string, just ensure that it is RCF compliant. + +At the end modified (or untouched) request should be emitted back to STDOUT, keeping original header, and hex-encoded. If you want to filter request, just not send it. Emitting responses back is required, even if you did not touch them. + +#### Advanced example +Imagine that you have auth system that randomly generate access tokens, which used later for accessing secure content. Since there is no pre-defined token value, naive approach without middleware (or if middleware use only request payloads) will fail, because replayed server have own tokens, not synced with origin. To fix this, our middleware should take in account responses of replayed and origin server, store `originalToken -> replayedToken` aliases and rewrite all requests using this token to use replayed alias. See [examples/middleware/token_modifier.go](https://github.com/buger/gor/tree/master/examples/middleware/token_modifier.go) and [middleware_test.go#TestTokenMiddleware](https://github.com/buger/gor/tree/master/middleware_test.go) as example of described scheme. + +*** + +You may also read about [[Request filtering]], [[Rate limiting]] and [[Request rewriting]]. \ No newline at end of file diff --git a/docs/Rate-limiting.md b/docs/Rate-limiting.md new file mode 100644 index 0000000..ce0003b --- /dev/null +++ b/docs/Rate-limiting.md @@ -0,0 +1,37 @@ +Rate limiting can be useful if you only want to forward parts of incoming traffic, for example, to not overload your test environment. There are two strategies: dropping random requests or dropping fractions of requests based on Header or URL param value. + +### Dropping random requests +Every input and output support random rate limiting. +There are two limiting algorithms: absolute or percentage based. + +**Absolute**: If for current second it reached specified requests limit - disregard the rest, on next second counter reset. + +**Percentage**: For input-file it will slowdown or speedup request execution, for the rest it will use the random generator to decide if request pass or not based on the chance you specified. + +You can specify your desired limit using the "|" operator after the server address, see examples below. + +#### Limiting replay using absolute number +``` +# staging.server will not get more than ten requests per second +gor --input-tcp :28020 --output-http "http://staging.com|10" +``` + +#### Limiting listener using percentage based limiter +``` +# replay server will not get more than 10% of requests +# useful for high-load environments +gor --input-raw :80 --output-tcp "replay.local:28020|10%" +``` + +### Consistent limiting based on Header or URL param value +If you have unique user id (like API key) stored in header or URL you can consistently forward specified percent of traffic only for the fraction of this users. +Basic formula looks like this: `FNV32-1A_hashing(value) % 100 >= chance`. Examples: +``` +# Limit based on header value +gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-header-limiter "X-API-KEY: 10%" + +# Limit based on header value +gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-param-limiter "api_key: 10%" +``` + +When limiting based on header or param only percentage based limiting supported. \ No newline at end of file diff --git a/docs/Replaying-HTTP-traffic.md b/docs/Replaying-HTTP-traffic.md new file mode 100644 index 0000000..7004323 --- /dev/null +++ b/docs/Replaying-HTTP-traffic.md @@ -0,0 +1,46 @@ +Gor can replay HTTP traffic using `--output-http` option: + +```bash +sudo ./gor --input-raw :8000 --output-http="http://staging.env" +``` + +You can [filter](Request filtering), [rate limit](Rate limiting) and [rewrite](Request rewriting) requests on the fly. + +### HTTP output workers +By default Gor creates a dynamic pool of workers: it starts with 10 and creates more HTTP output workers when the HTTP output queue length is greater than 10. The number of workers created (N) is equal to the queue length at the time which it is checked and found to have a length greater than 10. The queue length is checked every time a message is written to the HTTP output queue. No more workers will be spawned until that request to spawn N workers is satisfied. If a dynamic worker cannot process a message at that time, it will sleep for 100 milliseconds. If a dynamic worker cannot process a message for 2 seconds it dies. +You may specify fixed number of workers using `--output-http-workers=20` option. + +### Following redirects +By default Gor will ignore all redirects since they are handled by clients using your app, but in scenarios where your replayed environment introduces new redirects, you can enable them like this: +``` +gor --input-tcp replay.local:28020 --output-http http://staging.com --output-http-redirects 2 +``` +The given example will follow up to 2 redirects per request. + +### HTTP timeouts +By default http timeout for both request and response is 5 seconds. You can override it like this: +``` +gor --input-tcp replay.local:28020 --output-http http://staging.com --output-http-timeout 30s +``` + +### Response buffer +By default, to reduce memory consumption, internal HTTP client will fetch max 200kb of the response body (used if you use middleware), by you can increase limit using `--output-http-response-buffer` option (accepts number of bytes). + +### Basic Auth + +If your development or staging environment is protected by Basic Authentication then those credentials can be injected in during the replay: + +``` +gor --input-raw :80 --output-http "http://user:pass@staging.com" +``` + +Note: This will overwrite any Authorization headers in the original request. + + +### Multiple domains support + +If you app accepts traffic from multiple domains, and you want to keep original headers, there is specific `--http-original-host` with tells Gor do not touch Host header at all. + + +*** +You may also read about [[Saving and Replaying from file]] \ No newline at end of file diff --git a/docs/Request-filtering.md b/docs/Request-filtering.md new file mode 100644 index 0000000..07d2285 --- /dev/null +++ b/docs/Request-filtering.md @@ -0,0 +1,35 @@ +Filtering is useful when you need to capture only specific part of traffic, like API requests. It is possible to filter by URL, HTTP header or HTTP method. + +#### Allow url regexp +``` +# only forward requests being sent to the /api endpoint +gor --input-raw :8080 --output-http staging.com --http-allow-url /api +``` + +#### Disallow url regexp +``` +# only forward requests NOT being sent to the /api... endpoint +gor --input-raw :8080 --output-http staging.com --http-disallow-url /api +``` +#### Filter based on regexp of header + +``` +# only forward requests with an api version of 1.0x +gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^1\.0\d + +# only forward requests NOT containing User-Agent header value "Replayed by Gor" +gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor" +``` + +#### Filter based on HTTP method +Requests not matching a specified whitelist can be filtered out. For example to strip non-nullipotent requests: + +``` +gor --input-raw :80 --output-http "http://staging.server" \ + --http-allow-method GET \ + --http-allow-method OPTIONS +``` + + +----- +You may also read about [[Request rewriting]], [[Rate limiting]] and [[Middleware]] \ No newline at end of file diff --git a/docs/Request-rewriting.md b/docs/Request-rewriting.md new file mode 100644 index 0000000..adc68a4 --- /dev/null +++ b/docs/Request-rewriting.md @@ -0,0 +1,38 @@ +Gor supports rewriting of URLs, URL params and headers, see below. + +Rewriting may be useful if you test environment does not have the same data as your production, and you want to perform all actions in the context of `test` user: for example rewrite all API tokens to some test value. Other possible use cases are toggling features on/off using custom headers or rewriting URL's if they changed in the new environment. + +For more complex logic you can use [[Middleware]]. + +#### Rewrite URL based on a mapping +`--http-rewrite-url` expects value in ":" format: ":" is a dilimiter. In `` section you may use captured regexp group values. This works similar to `replace` method in Javascript or `gsub` in Ruby. + +``` +# Rewrites all `/v1/user//ping` requests to `/v2/user//ping` +gor --input-raw :8080 --output-http staging.com --http-rewrite-url /v1/user/([^\\/]+)/ping:/v2/user/$1/ping +``` + +#### Set URL param +Set request url param, if param already exists it will be overwritten. +``` +gor --input-raw :8080 --output-http staging.com --http-set-param api_key=1 +``` + +#### Set Header +Set request header, if header already exists it will be overwritten. May be useful if you need to identify requests generated by Gor or enable feature flagged functionality in an application: + +``` +gor --input-raw :80 --output-http "http://staging.server" \ + --http-header "User-Agent: Replayed by Gor" \ + --http-header "Enable-Feature-X: true" +``` + +#### Host header +Host header gets special treatment. By default Host get set to the value specified in --output-http. If you manually set --http-header "Host: anonther.com", Gor will not override Host value. + +If you app accepts traffic from multiple domains, and you want to keep original headers, there is specific `--http-original-host` with tells Gor do not touch Host header at all. + + +*** + +You may also read about [[Request filtering]], [[Rate limiting]] and [[Middleware]] \ No newline at end of file diff --git a/docs/Running-as-non-root-user.md b/docs/Running-as-non-root-user.md new file mode 100644 index 0000000..66c9512 --- /dev/null +++ b/docs/Running-as-non-root-user.md @@ -0,0 +1,17 @@ +You can enable Gor for non-root users in a secure method by using the following commands + +``` +# Following commands assume that you put `gor` binary to /usr/local/bin +add gor +addgroup gor +chgrp gor /usr/local/bin/gor +chmod 0750 /usr/local/bin/gor +setcap "cap_net_raw,cap_net_admin+eip" /usr/local/bin/gor +``` + +As a brief explanation of the above. +* We create a group called gor. +* We then add the user you want to the new group so they will be able to use gor without sudo +* We then change the user/group of gor binary the new group. +* We then make sure the permissions are set on gor binary so that members of the group can execute it but other normal users cannot. +* We then use `setcap` to give the CAP_NET_RAW and CAP_NET_ADMIN privilege to the executable when it runs. This is so that Gor can open its raw socket which is not normally permitted unless you are root. \ No newline at end of file diff --git a/docs/Saving-and-Replaying-from-file.md b/docs/Saving-and-Replaying-from-file.md new file mode 100644 index 0000000..568c7d5 --- /dev/null +++ b/docs/Saving-and-Replaying-from-file.md @@ -0,0 +1,105 @@ +You can save requests to file, and replay them later. While replaying it will preserve the original time differences between requests. If you apply [percentage based limiting](Rate Limiting) timing between requests will be reduced or increased appropriately: this approach opens possibilities like load testing, see below. + +```bash +# write to file +gor --input-raw :80 --output-file requests.log + +# read from file +gor --input-file requests.gor --output-http "http://staging.com" +``` + +By default Gor writes files in chunks. This configurable using `--output-file-append` option: the flushed chunk is appended to existence file or not. The default is **false**. By default, `--output-file` flushes each chunk to a different path. + +```bash +gor ... --output-file %Y%m%d.log +# append false +20140608_0.log +20140608_1.log +20140609_0.log +20140609_1.log +``` + +This makes parallel file processing easy. But if you want to disable this behavior, you can disable it by adding `--output-file-append` option: + +```bash +gor ... --output-file %Y%m%d.log --output-file-append +# append true +20140608.log +20140609.log +``` + +If you run gor multiple times, and it finds existing files, it will continue from last known index. + +### Chunk size + +You can set chunk limits using `--output-file-size-limit` and `--output-file-queue-limit` options. +The length of the chunk queue and the size of each chunk, respectively. The default values are 256 and 32mb, respectively. The suffixes “k” (KB), “m” (MB), and “g” (GB) can be used for `output-file-size-limit`. +If you want to have only size constraint, you can set `--output-file-queue-limit` to 0, and vice versa. + +```bash +gor --input-raw :80 --output-file %Y-%m-%d.gz --output-file-size-limit 256m --output-file-queue-limit 0 +``` + +### Using date variables in file names +For example, you can tell to create new file each hour: `--output-file /mnt/logs/requests-%Y-%m-%d-%H.log` +It will create new file for each hour: requests-2016-06-01-12.log, requests-2016-06-01-13.log, ... + +The time format used as part of the file name. The following characters are replaced with actual values when the file is created: + +* `%Y`: year including the century (at least 4 digits) +* `%m`: month of the year (01..12) +* `%d`: Day of the month (01..31) +* `%H`: Hour of the day, 24-hour clock (00..23) +* `%M`: Minute of the hour (00..59) +* `%S`: Second of the minute (00..60) + +The default format is `%Y%m%d%H`, which creates one file per hour. + + +### GZIP compression +To read or write GZIP compressed files ensure that file extension ends with ".gz": `--output-file log.gz` + +### Replaying from multiple files + +`--input-file` accepts file pattern, for example: `--input-file logs-2016-05-*`: it will replay all the files, sorting them in lexicographical order. + +### Buffered file output +Gor has memory buffer when it writes to file, and continuously flush changes to the file. Flushing to file happens if the buffer is filled, forced flush every 1 second, or if Gor is closed. You can change it using `--output-file-flush-interval` option. It most cases it should not be touched. + +### File format +HTTP requests stored as it is, plain text: headers and bodies. Requests separated by `\n🐵🙈🙉\n` line (using such sequence for uniqueness and fun). Before each request goes single line with meta information containing payload type (1 - request, 2 - response, 3 - replayed response), unique request ID (request and response have the same) and timestamp when request was made. An example of 2 requests: + +``` +1 d7123dasd913jfd21312dasdhas31 127345969\n +GET / HTTP/1.1\r\n +\r\n +\n +🐵🙈🙉 +\n +POST /upload HTTP/1.1\r\n +Content-Length: 7\r\n +Host: www.w3.org\r\n +\r\n +a=1&b=2 +``` +Note that technically \r and \n symbols are invisible, and indicate new lines. I made them visible in example just to show how it looks on byte level. + +Making it text friendly allows writing simple parsers and use console tools like `grep` to do an analysis. You can even edit them manually, but be sure that your file editor does not change line endings. + +## Performance testing + +Currently, this functionality supported only by `input-file` and only when using percentage based limiter. Unlike default limiter for `input-file` instead of dropping requests it will slowdown or speedup request emitting. Note that **limiter is applied to input**: + +``` +# Replay from file on 2x speed +gor --input-file "requests.gor|200%" --output-http "staging.com" +``` + +Use `--stats --output-http-stats` to see latency stats. + +### Looping files for replaying indefinitely +You can loop the same set of files, so when the last one replays all the requests, it will not stop, and will start from first one again. Having the only small amount of requests you can do extensive performance testing. +Pass `--input-file-loop` to make it work. + +*** +You may also read about [[Capturing and replaying traffic]] and [[Rate limiting]] \ No newline at end of file diff --git a/docs/The-Basics.md b/docs/The-Basics.md new file mode 100644 index 0000000..799c48c --- /dev/null +++ b/docs/The-Basics.md @@ -0,0 +1,17 @@ +### Overview +Gor architecture tries to follow UNIX philosophy: everything made of pipes, various inputs multiplexing data to outputs. + +You can [rate limit](Rate limiting), [filter](Request filtering), [rewrite](Request rewriting) requests or even use your own [middleware](Middleware) to implement custom logic. Also, it is possible to replay requests at the higher rate for [load testing](Saving and Replaying from file). + +### Available input and output plugins + +Available inputs: + * `--input-raw` - used to capture HTTP traffic, you should specify IP address or interface and application port. More about [[Capturing and replaying traffic]]. + * `--input-file` - accepts file which previously was recorded using `--output-file`. More about [[Saving and Replaying from file]] + * `--input-tcp` - used by Gor aggregation instance if you decided forward traffic from multiple forwarder Gor instances to it. Read about using [[Aggregator-forwarder setup]]. + +Available outputs: + * `--output-http` - replay HTTP traffic to given endpoint, accepts base url. Read [more about it](Replaying HTTP traffic) + * `--output-file` - records incoming traffic to the file. More about [[Saving and Replaying from file]] + * `--output-tcp` - forward incoming data to another Gor instance, used in conjunction with `--input-tcp`. Read more about [[Aggregator-forwarder setup]]. + * `--output-stdout` - used for debugging, outputs all data to stdout. \ No newline at end of file diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..e473071 --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,90 @@ +Gor can report stats on the `output-tcp` and `output-http` request queues. Stats are reported to the console every 5 seconds in the form `latest,mean,max,count,count/second` by using the `--output-http-stats` and `--output-tcp-stats` options. + +Examples: + +``` +2014/04/23 21:17:50 output_tcp:latest,mean,max,count,count/second +2014/04/23 21:17:50 output_tcp:0,0,0,0,0 +2014/04/23 21:17:55 output_tcp:1,1,2,68,13 +2014/04/23 21:18:00 output_tcp:1,1,2,92,18 +2014/04/23 21:18:05 output_tcp:1,1,2,119,23 +``` + +``` +2014/04/23 21:19:46 output_http:latest,mean,max,count,count/second +2014/04/23 21:19:46 output_http:0,0,0,0,0 +2014/04/23 21:19:51 output_http:0,0,0,0,0 +2014/04/23 21:19:56 output_http:0,0,0,0,0 +2014/04/23 21:20:01 output_http:1,0,1,50,10 +2014/04/23 21:20:06 output_http:1,1,4,72,14 +2014/04/23 21:20:11 output_http:1,0,1,179,35 +2014/04/23 21:20:16 output_http:1,0,1,148,29 +2014/04/23 21:20:21 output_http:1,1,2,91,18 +2014/04/23 21:20:26 output_http:1,1,2,150,30 +2014/04/23 21:18:15 output_http:100,99,100,70,14 +2014/04/23 21:18:21 output_http:100,99,100,55,11 +``` + +### How can I tell if I have bottlenecks? +Key areas that sometimes experience bottlenecks are the output-tcp and output-http functions which have internal queues for requests. Each queue has an upper limit of 100. Enable stats reporting to see if any queues are experiencing bottleneck behavior. + +#### Output HTTP bottlenecks +When running a Gor replay the output-http feature may bottleneck if: + + * the replay has inadequate bandwidth. If the replay is receiving or sending more messages than its network adapter can handle the output-http-stats may report that the output-http queue is filling up. See if there is a way to upgrade the replay's bandwidth. + * with `--output-http-workers` set to anything other than `-1` the `-output-http` target is unable to respond to messages in a timely manner. The http output workers which take messages off the output-http queue, process the request, and ensure that the request did not result in an error may not be able to keep up with the number of incoming requests. If the replay is not using dynamic worker scaling (`--output-http-workers=-1`) The optimal number of output-http-workers can be determined with the formula `output-workers = (Average number of requests per second)/(Average target response time per second)`. + +#### Output TCP bottlenecks +When using the Gor listener the output-tcp feature may bottleneck if: + + * the replay is unable to accept and process more requests than the listener is able generate. Prior to troubleshooting the output-tcp bottleneck, ensure that the replay target is not experiencing any bottlenecks. + * the replay target has inadequate bandwidth to handle all its incoming requests. If a replay target's incoming bandwidth is maxed out the output-tcp-stats may report that the output-tcp queue is filling up. See if there is a way to upgrade the replay's bandwidth. + + +#### Tuning + +To achieve the top most performance you should tune the source server system limits: + + net.ipv4.tcp_max_tw_buckets = 65536 + net.ipv4.tcp_tw_recycle = 1 + net.ipv4.tcp_tw_reuse = 0 + net.ipv4.tcp_max_syn_backlog = 131072 + net.ipv4.tcp_syn_retries = 3 + net.ipv4.tcp_synack_retries = 3 + net.ipv4.tcp_retries1 = 3 + net.ipv4.tcp_retries2 = 8 + net.ipv4.tcp_rmem = 16384 174760 349520 + net.ipv4.tcp_wmem = 16384 131072 262144 + net.ipv4.tcp_mem = 262144 524288 1048576 + net.ipv4.tcp_max_orphans = 65536 + net.ipv4.tcp_fin_timeout = 10 + net.ipv4.tcp_low_latency = 1 + net.ipv4.tcp_syncookies = 0 +*** + +### Gor is crashing with following stacktrace +``` +fatal error: unexpected signal during runtime execution +[signal 0xb code=0x1 addr=0x63 pc=0x7ffcdfdf8b2c] + +runtime stack: +runtime.throw(0xad8380, 0x2a) + /usr/local/go/src/runtime/panic.go:547 +0x90 +runtime.sigpanic() + /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a + +goroutine 103 [syscall, locked to thread]: +runtime.cgocall(0x7b35a0, 0xc82121f1e8, 0x0) + /usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc82121f188 sp=0xc82121f158 +net._C2func_getaddrinfo(0x7ffcec0008c0, 0x0, 0xc821b221e0, 0xc8217b2b18, 0x0, 0x0, 0x0) + ??:0 +0x55 fp=0xc82121f1e8 sp=0xc82121f188 +net.cgoLookupIPCNAME(0x7fffb17208ab, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb17200) +``` + +There is a chance that you hit Go bug. The crash comes from the CGO version of DNS resolver. +By default Go based version used, but ins some cases [it switches to CGO based](https://golang.org/pkg/net/#hdr-Name_Resolution). It is possible to force Go based DNS resolver using GODEBUG environment variable: +`sudo GODEBUG="netdns=go" ./gor --input-raw :80 --output-http staging.env` + + + +Also, see [[FAQ]] \ No newline at end of file diff --git a/docs/[PRO]-Recording-and-replaying-keep-alive-TCP-sessions.md b/docs/[PRO]-Recording-and-replaying-keep-alive-TCP-sessions.md new file mode 100644 index 0000000..cea7702 --- /dev/null +++ b/docs/[PRO]-Recording-and-replaying-keep-alive-TCP-sessions.md @@ -0,0 +1,11 @@ +> **This feature available only in PRO version. See https://goreplay.org/pro.html for details.** + +By default, GoReplay does not guarantee that when you record keep-alive TCP session, it will be replayed in the same TCP connection as well. This is ok for most of the cases, but it does not give an accurate number of TCP sessions while replaying, also may cause issues if your application state depends on TCP session (do not mess with HTTP session). + +[GoReplay PRO](https://goreplay.org/pro.html) extension adds support for accurate recording and replaying of keep-alive TCP sessions. Separate connection to your server is created per original session and it makes benchmarks and tests incredibly accurate. To enable session recognition you just need to pass `--recognize-tcp-sessions` option. + +``` +gor --input-raw :80 --recognize-tcp-sessions --output-http http://test.target +``` + +Note that enabling this option also change algorithm of distributing traffic when using `--split-output`, see [Distributed configuration]. \ No newline at end of file diff --git a/docs/[PRO]-Replaying-Binary-protocols.md b/docs/[PRO]-Replaying-Binary-protocols.md new file mode 100644 index 0000000..7709d9a --- /dev/null +++ b/docs/[PRO]-Replaying-Binary-protocols.md @@ -0,0 +1,14 @@ +> **This feature available only in PRO version. See https://goreplay.org/pro.html for details.** + +Gor includes basic support for working with binary formats like `thrift` or `protocol-buffers`. To start set `--input-raw-protocol` to 'binary' (by default 'http'). For replaying, you should use `--output-binary`, example: + +``` +gor --input-raw :80 --input-raw-protocol binary --output-binary staging:8081 +``` + +While working with `--input-raw` you may notice a 2-second delay before messages are emitted to the outputs. This behaviour is expected and happening because for general binary protocol it is impossible to know when TCP message ends, so Gor has to set inactivity timeout. Each protocol has own rules (for example write message length as first bytes), and require individual handling to know message length. We consider improving detailed protocol support for `thrift`, `protocol-buffer` and etc. + +Note that you can use all load testing features for binary protocols. For example, the following command will loop and replay recorded payload on 10x speed for 30 seconds: +``` +gor --input-file './binary*.gor|1000%' --output-binary staging:9091 --input-file-loop --exit-after 30s +``` \ No newline at end of file diff --git a/docs/_Footer.md b/docs/_Footer.md new file mode 100644 index 0000000..3b97cd2 --- /dev/null +++ b/docs/_Footer.md @@ -0,0 +1 @@ +[Website](https://goreplay.org) | [PRO version](https://goreplay.org/pro.html) | [[Getting started]] | [[FAQ]] | [Join newsletter](https://www.getdrip.com/forms/89690474/submissions/new) \ No newline at end of file