github/minio
4
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-04-21 12:41:46 +00:00
Code Issues Releases Wiki Activity
7,655 Commits 6 Branches 471 Tags
ad511b0eb8a928fd79d0d995e227b8cdfeb4f736
Commit Graph
7655 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Klaus PostandGitHub ad511b0eb8 tests: Fix occasional data race (#11223) 
CI tests could trigger a data race.

Servers are generally not expected to reinitialize, so tests could trigger data races when reinitializing and async operations are running.

We add the option to safely reset global vars instead of overwriting.

Fixes races like:

```
WARNING: DATA RACE
Read at 0x00000477ab18 by goroutine 1159:
  github.com/minio/minio/cmd.FileInfo.ToObjectInfo()
      /home/runner/work/minio/minio/cmd/erasure-metadata.go:105 +0x16d
  github.com/minio/minio/cmd.erasureObjects.putObject()
      /home/runner/work/minio/minio/cmd/erasure-object.go:748 +0x13f8
  github.com/minio/minio/cmd.(*erasureObjects).listPath.func3.2()
      /home/runner/work/minio/minio/cmd/metacache-set.go:682 +0x7d3
  github.com/minio/minio/cmd.newMetacacheBlockWriter.func1.2()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:777 +0x1c4
  github.com/minio/minio/cmd.newMetacacheBlockWriter.func1()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:806 +0x614

Previous write at 0x00000477ab18 by goroutine 1269:
  [failed to restore the stack]

Goroutine 1159 (running) created at:
  github.com/minio/minio/cmd.newMetacacheBlockWriter()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:760 +0x112
  github.com/minio/minio/cmd.(*erasureObjects).listPath.func3()
      /home/runner/work/minio/minio/cmd/metacache-set.go:672 +0xe22

Goroutine 1269 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1095 +0x537
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1339 +0xa6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1050 +0x1eb
  testing.runTests()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1337 +0x594
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1252 +0x2ff
  github.com/minio/minio/cmd.TestMain()
      /home/runner/work/minio/minio/cmd/test-utils_test.go:120 +0x44e
  main.main()
      _testmain.go:1408 +0x223
==================
==================
WARNING: DATA RACE
Read at 0x00000477aae8 by goroutine 1159:
  github.com/minio/minio/cmd.(*BucketVersioningSys).Enabled()
      /home/runner/work/minio/minio/cmd/bucket-versioning.go:26 +0x52
  github.com/minio/minio/cmd.FileInfo.ToObjectInfo()
      /home/runner/work/minio/minio/cmd/erasure-metadata.go:105 +0x197
  github.com/minio/minio/cmd.erasureObjects.putObject()
      /home/runner/work/minio/minio/cmd/erasure-object.go:748 +0x13f8
  github.com/minio/minio/cmd.(*erasureObjects).listPath.func3.2()
      /home/runner/work/minio/minio/cmd/metacache-set.go:682 +0x7d3
  github.com/minio/minio/cmd.newMetacacheBlockWriter.func1.2()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:777 +0x1c4
  github.com/minio/minio/cmd.newMetacacheBlockWriter.func1()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:806 +0x614

Previous write at 0x00000477aae8 by goroutine 1269:
  [failed to restore the stack]

Goroutine 1159 (running) created at:
  github.com/minio/minio/cmd.newMetacacheBlockWriter()
      /home/runner/work/minio/minio/cmd/metacache-stream.go:760 +0x112
  github.com/minio/minio/cmd.(*erasureObjects).listPath.func3()
      /home/runner/work/minio/minio/cmd/metacache-set.go:672 +0xe22

Goroutine 1269 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1095 +0x537
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1339 +0xa6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1050 +0x1eb
  testing.runTests()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1337 +0x594
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.14.13/x64/src/testing/testing.go:1252 +0x2ff
  github.com/minio/minio/cmd.TestMain()
      /home/runner/work/minio/minio/cmd/test-utils_test.go:120 +0x44e
  main.main()
      _testmain.go:1408 +0x223
==================
```
 2021-01-05 10:45:26 -08:00
HarshavardhanaandGitHub cb0eaeaad8 feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY (#11185) 2021-01-05 10:22:57 -08:00
Minio Trusted f3f0041ad0 Update yaml files to latest version RELEASE.2021-01-05T05-22-38Z 2021-01-05 05:42:54 +00:00
HarshavardhanaandGitHub d0027c3c41 do not use large buffers if not necessary (#11220) 
without this change, there is a performance
regression for small objects GETs, this makes
the overall speed to go back to pre '59d363'
commit days.
RELEASE.2021-01-05T05-22-38Z 		2021-01-04 18:51:52 -08:00
Anis ElleuchandGitHub cb7fc99368 handlers: Avoid initializing a struct in each handler call (#11217) 2021-01-04 09:54:22 -08:00
HarshavardhanaandGitHub a4383051d9 remove/deprecate crawler disable environment (#11214) 
with changes present to automatically throttle crawler
at runtime, there is no need to have an environment
value to disable crawling. crawling is a fundamental
piece for healing, lifecycle and many other features
there is no good reason anyone would need to disable
this on a production system.

* Apply suggestions from code review
 2021-01-04 09:43:31 -08:00
HarshavardhanaandGitHub e7ae49f9c9 fix: calculate prometheus disks_offline/disks_total correctly (#11215) 
fixes #11196
 2021-01-04 09:42:09 -08:00
Anis ElleuchGitHubAnis Elleuch
 		153d4be032 tracing: NumSubscribers() to use atomic instead of mutex (#11219) 
globalSubscribers.NumSubscribers() is heavily used in S3 requests and it
uses mutex, use atomic.Load instead since it is faster

Co-authored-by: Anis Elleuch <anis@min.io>
 2021-01-04 09:40:30 -08:00
Anis ElleuchandGitHub dfd99b6d8f handlers: Little bit more optimizations (#11211) 2021-01-04 00:01:06 -08:00
HarshavardhanaandGitHub c4b1d394d6 erasure: avoid io.Copy in hotpaths to reduce allocation (#11213) 2021-01-03 16:27:34 -08:00
HarshavardhanaandGitHub c4131c2798 feat: Small object optimization read data in single bulk call (#11207) 2021-01-03 11:27:57 -08:00
Anis ElleuchandGitHub c9d502e6fa parentDirIsObject() to return quickly with inexistant parent (#11204) 
Rewrite parentIsObject() function. Currently if a client uploads
a/b/c/d, we always check if c, b, a are actual objects or not.

The new code will check with the reverse order and quickly quit if 
the segment doesn't exist.

So if a, b, c in 'a/b/c' does not exist in the first place, then returns
false quickly.
 2021-01-02 12:01:29 -08:00
Anis ElleuchandGitHub 677e80c0f8 xl: Remove check-dir in ReadVersion (#11200) 
The only purpose of check-dir flag in
ReadVersion is to return 404 when
an object has xl.meta but without data.

This is causing an extract call to the disk 
which can be penalizing in case of busy system
where disks receive many concurrent access.
 2021-01-02 10:35:57 -08:00
Harshavardhana aa85af4d1a fix: missing CopyObjectPart maxClients reorder 2021-01-01 23:07:37 -08:00
Anis ElleuchandGitHub ae731d232f trace: Reorder http/trace maxClients wrapping for correct tracing (#11202) 
mc admin trace does not show the correct handler name in the output: it
is printing `maxClients' for all handlers. The reason is that the wrong
order of handler wrapping.
 2021-01-01 23:06:07 -08:00
Anis ElleuchandGitHub a317d220ed xl-storage: Do not stat bucket assuming the object exists (#11201) 
In HEAD/GET, only STAT the bucket if the 
object does not exist to return the correct
error response.
 2021-01-01 09:44:36 -08:00
HarshavardhanaandGitHub 3e1221a01c fix: log once updating dataUsageCache versions (#11190) 
also reduce usage of *bytes.Buffer for
reading `usage-cache.bin`
 2020-12-31 09:45:09 -08:00
Baptiste Mille-MathiasandGitHub c1f6ca6697 Fix caddy project url (#11198) 2020-12-31 09:44:07 -08:00
Ritesh H ShuklaandGitHub 36fc2f98ed fix: admin trace throttled requests (#11192) 2020-12-30 21:04:55 -08:00
Ritesh H ShuklaandGitHub 556524c715 Reduce logging when peer is offline (#11184) 2020-12-30 14:38:54 -08:00
Harshavardhana 428f288379 update release Dockerfile string 2020-12-30 08:50:43 -08:00
0xflotusandGitHub cde801282d chore: enabled syntax highlighting in docs (#11182) 2020-12-29 17:38:28 -08:00
Ravind KumarandGitHub 6cf0008469 fix: docs typo in object lock docs (#11181) 2020-12-29 16:14:10 -08:00
Minio Trusted 7b0330a98c Update yaml files to latest version RELEASE.2020-12-29T23-29-29Z 2020-12-29 23:46:24 +00:00
HarshavardhanaandGitHub cc457f1798 fix: enhance logging in crawler use console.Debug instead of logger.Info (#11179) RELEASE.2020-12-29T23-29-29Z 2020-12-29 01:57:28 -08:00
HarshavardhanaandGitHub ca0d31b09a fix: re-arrange handlers to handle requests on /minio (#11177) 
fixes #11175
 2020-12-28 17:10:33 -08:00
HarshavardhanaandGitHub 445a9bd827 fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) 
Fixes two problems

- Double healing when bitrot is enabled, instead heal attempt
  once in applyActions() before lifecycle is applied.

- If applyActions() is successful and getSize() returns proper
  value, then object is accounted for and should be removed
  from the oldCache namespace map to avoid double heal attempts.
 2020-12-28 10:31:00 -08:00
HarshavardhanaandGitHub d8d25a308f fix: use HealObject for cleaning up dangling objects (#11171) 
main reason is that HealObjects starts a recursive listing
for each object, this can be a really really long time on
large namespaces instead avoid recursive listing just
perform HealObject() instead at the prefix.

delete's already handle purging dangling content, we
don't need to achieve this by doing recursive listing,
this in-turn can delay crawling significantly.
 2020-12-27 15:42:20 -08:00
HarshavardhanaandGitHub c19e6ce773 avoid a crash in crawler when lifecycle is not initialized (#11170) 
Bonus for static buffers use bytes.NewReader instead of
bytes.NewBuffer, to use a more reader friendly implementation
 2020-12-26 22:58:06 -08:00
Minio Trusted d3c853a3be Update yaml files to latest version RELEASE.2020-12-26T01-35-54Z 2020-12-26 01:53:30 +00:00
HarshavardhanaandGitHub 59d3639396 fix: inherit heal opts globally, including bitrot settings (#11166) 
Bonus re-use ReadFileStream internal io.Copy buffers, fixes
lots of chatty allocations when reading metacache readers
with many sustained concurrent listing operations

```
   17.30GB  1.27% 84.80%    35.26GB  2.58%  io.copyBuffer
```
RELEASE.2020-12-26T01-35-54Z 		2020-12-24 23:04:03 -08:00
HarshavardhanaandGitHub 027e17468a fix: discarding results do not attempt in-memory metacache writer (#11163) 
Optimizations include

- do not write the metacache block if the size of the
  block is '0' and it is the first block - where listing
  is attempted for a transient prefix, this helps to
  avoid creating lots of empty metacache entries for
  `minioMetaBucket`

- avoid the entire initialization sequence of cacheCh
  , metacacheBlockWriter if we are simply going to skip
  them when discardResults is set to true.

- No need to hold write locks while writing metacache
  blocks - each block is unique, per bucket, per prefix
  and also is written by a single node.
 2020-12-24 15:02:02 -08:00
HarshavardhanaandGitHub 45ea161f8d webUI: change listing to 1000 keys from browser UI (#11159) 
gateway implementations do not handle maxKeys being
`-1` properly unlike MinIO implementation, handle it
by setting an appropriate value.

fixes #11158
 2020-12-23 19:58:15 -08:00
Poorna KrishnamoorthyandGitHub 7b8a456f68 Update lifecycle README docs (#11160) 
Removing reference to transition feature in docs as this
feature is being revamped to provide better extensibility
across different cloud targets.
 2020-12-23 19:56:55 -08:00
Harshavardhana b43906f6ee fix: docs typos and keywords 2020-12-23 11:59:20 -08:00
HarshavardhanaandGitHub 6a66f142d4 fix: strict quorum in list should list on all drives (#11157) 
current implementation was incorrect, it in-fact
assumed only read quorum number of disks. in-fact
that value is only meant for read quorum good entries
from all online disks.

This PR fixes this behavior properly.
 2020-12-23 09:26:40 -08:00
HarshavardhanaandGitHub 5982965839 fix: re-use bytes.Buffer using sync.Pool (#11156) 2020-12-22 23:22:37 -08:00
Minio Trusted bfb92a27b7 Update yaml files to latest version RELEASE.2020-12-23T02-24-12Z 2020-12-23 02:43:25 +00:00
Harshavardhana 8565cefe4e fix: allow HTTP2.0 to be always configured RELEASE.2020-12-23T02-24-12Z 2020-12-22 16:32:58 -08:00
Andreas AuernhammerandGitHub 8cdf2106b0 refactor cmd/crypto code for SSE handling and parsing (#11045) 
This commit refactors the code in `cmd/crypto`
and separates SSE-S3, SSE-C and SSE-KMS.

This commit should not cause any behavior change
except for:
  - `IsRequested(http.Header)`

which now returns the requested type {SSE-C, SSE-S3,
SSE-KMS} and does not consider SSE-C copy headers.

However, SSE-C copy headers alone are anyway not valid.
 2020-12-22 09:19:32 -08:00
HarshavardhanaandGitHub 35fafb837b fix: issues with handling delete markers in metacache (#11150) 
Additional cases handled

- fix address situations where healing is not
  triggered on failed writes and deletes.

- consider object exists during listing when
  metadata can be successfully decoded.
 2020-12-22 09:16:43 -08:00
HarshavardhanaandGitHub 274bbad5cb fix: select always online peers for remote listing (#11153) 
always find the right set of online peers for remote listing,
this may have an effect on listing if the server is down - we
should do this to avoid always performing transient operations
on bucket->peerClient that is permanently or down for a long
period.
 2020-12-22 09:16:07 -08:00
HarshavardhanaandGitHub 5c451d1690 update x/net/http2 to address few bugs (#11144) 
additionally also configure http2 healthcheck
values to quickly detect unstable connections
and let them timeout.

also use single transport for proxying requests
 2020-12-21 21:42:38 -08:00
Poorna KrishnamoorthyGitHubPoorna Krishnamoorthy
 		c987313431 Encrypt remote target if kms is configured (#11034) 
Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2020-12-21 16:21:33 -08:00
Anis ElleuchandGitHub 2ecaab55a6 admin: ServerInfo returns info without object layer initialized (#11142) 2020-12-21 09:35:19 -08:00
HarshavardhanaandGitHub 3e792ae2a2 fix: change defaults for DNS cache dialer (#11145) 2020-12-21 09:33:29 -08:00
Yingrong ZhaoandGitHub 6df6ac0f34 fix testMultipartUploadFailure to properly cleanup (#11137) 2020-12-21 08:23:27 -08:00
HarshavardhanaandGitHub 4cc500a041 normalize users with double // in accessKeys (#11143) 
Bonus fix, use constant time compare for secret keys  in web-handlers.go:SetAuth()
 2020-12-20 10:09:51 -08:00
HarshavardhanaandGitHub d8e28830cf fix: allow STS creds for admin accounts to add users (#11138) 
Allow rotating creds with privileges to add users

fixes https://github.com/minio/console/issues/529
 2020-12-19 13:24:21 -08:00
HarshavardhanaandGitHub 3e16ec457a fix: support user/groups with '/' character (#11127) 
NOTE: user/groups with `//` shall be normalized to `/`

fixes #11126
 2020-12-19 09:36:37 -08:00