Commit Graph
2804 Commits
Author SHA1 Message Date
KevinSmileandGitHub 0ebb73ee2e use const instead of literals (#10292) 2020-08-19 16:43:52 -07:00
HarshavardhanaandGitHub c8b84a0e9e Add nancy vulnerability scanner (#10289) 2020-08-19 14:25:21 -07:00
Ritesh H ShuklaandGitHub 3acb5cff45 Update code comment (#10287) 2020-08-19 14:24:58 -07:00
HarshavardhanaandGitHub 74116204ce handle fresh setup with mixed drives (#10273)
fresh drive setups when one of the drive is
a root drive, we should ignore such a root
drive and not proceed to format.

This PR handles this properly by marking
the disks which are root disk and they are
taken offline.
2020-08-18 14:37:26 -07:00
HarshavardhanaandGitHub e4a44f6224 fix: commonPrefixes behavior in ListObjectVersions (#10286)
```
$ aws s3api --profile minio --endpoint-url http://localhost:9003 \
    list-object-versions --bucket testbucket \
    --delimiter / --prefix Veeam/Archive/

{
    "CommonPrefixes": [
        {
            "Prefix": "Veeam/Archive/003/"
        }
    ]
}
```

Also add coverage tests similar to ListObjects to
catch errors in future, skip these tests in FS mode
2020-08-18 12:19:44 -07:00
poornasandGitHub 0272973175 Fix regression in web ui for retention (#10285)
Fixes: #10283 regression from PR #9259
2020-08-18 12:09:42 -07:00
HarshavardhanaandGitHub d2a3f92452 fix: health handler for lockers (#10280) 2020-08-18 07:27:41 -07:00
HarshavardhanaandGitHub ede86845e5 docs: Add policy variables for resource and conditions (#10278)
Bonus fix adds LDAP policy variable and clarifies the
usage of policy variables for temporary credentials.

fixes #10197
2020-08-17 17:39:55 -07:00
HarshavardhanaandGitHub e57c742674 use single dynamic timeout for most locked API/heal ops (#10275)
newDynamicTimeout should be allocated once, in-case
of temporary locks in config and IAM we should
have allocated timeout once before the `for loop`

This PR doesn't fix any issue as such, but provides
enough dynamism for the timeout as per expectation.
2020-08-17 11:29:58 -07:00
Klaus PostandGitHub bb5976d727 healbucket: Send object version ID (#10263)
Based on our previous conversations I assume we should send the version
 id when healing an object.

Maybe we should even list object versions and heal all?
2020-08-17 08:25:44 -07:00
HarshavardhanaandGitHub f7c1a59de1 add validation logs for configured Logger/Audit HTTP targets (#10274)
extra logs in-case of misconfiguration of audit/logger targets
2020-08-16 10:25:00 -07:00
Anis ElleuchandGitHub 51ba1dac49 listing: Fix result when prefix is an object with a slash (#10267)
In a non recursive mode, issuing a list request where prefix
is an existing object with a slash and delimiter is a slash will
return entries in the object directory (data dir IDs)

```
$ aws s3api --profile minioadmin --endpoint-url http://localhost:9000 \
        list-objects-v2 --bucket testbucket --prefix code_of_conduct.md/ --delimiter '/'
{
    "CommonPrefixes": [
        {
            "Prefix":
"code_of_conduct.md/ec750fe0-ea7e-4b87-bbec-1e32407e5e47/"
        }
    ]
}
```

This commit adds a fast exit track in Walk() in this specific case.
2020-08-14 20:13:24 -07:00
HarshavardhanaandGitHub a4463dd40f fix: storageClass shouldn't set the value upon failure (#10271) 2020-08-14 19:48:04 -07:00
HarshavardhanaandGitHub 83a82d818e allow lock tolerance to match storage-class drive tolerance (#10270) 2020-08-14 18:17:14 -07:00
HarshavardhanaandGitHub 1d1c4430b2 decrypt ETags in parallel around 500 at a time (#10261)
Listing speed-up gained from 10secs for
just 400 entries to 2secs for 400 entries
2020-08-14 11:56:35 -07:00
HarshavardhanaandGitHub 43e6d1ce2d fix: missing proxy request by bucket for ListVersions (#10260) 2020-08-13 16:31:58 -07:00
HarshavardhanaandGitHub 30da442a85 rootDisk on containers can have different device Id (#10259)
use `/etc/hosts` instead of `/` to check for common
device id, if the device is same for `/etc/hosts`
and the --bind mount to detect root disks.

Bonus enhance healthcheck logging by adding maintenance
tags, for all messages.
2020-08-13 15:21:20 -07:00
HarshavardhanaandGitHub 038d91feaa fix: add public certs automatically as part of global CAs (#10256) 2020-08-13 09:46:50 -07:00
HarshavardhanaandGitHub e7ba78beee use GlobalContext instead of context.Background when possible (#10254) 2020-08-13 09:16:01 -07:00
Harshavardhana b32d0a5b60 use the correct endpoints for offline drives 2020-08-12 19:17:49 -07:00
poornasandGitHub 79e21601b0 fix: web handlers to enforce replication (#10249)
This PR also preserves source ETag for replication
2020-08-12 17:32:24 -07:00
HarshavardhanaandGitHub 34253aa595 feat: cache env value in-case network is not reachable (#10251) 2020-08-12 16:53:15 -07:00
HarshavardhanaandGitHub 79ed7ce451 fs: listObjects shouldn't take FS locks while listing (#10248) 2020-08-12 15:23:14 +05:30
HarshavardhanaandGitHub 0dd3a08169 move the certPool loader function into pkg/certs (#10239) 2020-08-11 08:29:50 -07:00
Klaus PostandGitHub f8f290e848 security: Remove insecure custom headers (#10244)
Background: https://github.com/google/security-research/security/advisories/GHSA-76wf-9vgp-pj7w

Remove these custom headers from incoming and outgoing requests.
2020-08-11 08:29:29 -07:00
HarshavardhanaandGitHub 1e2ebc9945 feat: time to bring back http2.0 support (#10230)
Bonus move our CI/CD to go1.14
2020-08-10 09:02:29 -07:00
HarshavardhanaandGitHub 2a9819aff8 fix: refactor background heal for cluster health (#10225) 2020-08-07 19:43:06 -07:00
HarshavardhanaandGitHub 6c6137b2e7 add cluster maintenance healthcheck drive heal affinity (#10218) 2020-08-07 13:22:53 -07:00
Anis ElleuchandGitHub 9138b2b503 Avoid duplicate headers when proxying S3 listing requests (#10220) 2020-08-07 04:10:16 -07:00
HarshavardhanaandGitHub 77509ce391 Support looking up environment remotely (#10215)
adds a feature where we can fetch the MinIO
command-line remotely, this
is primarily meant to add some stateless
nature to the MinIO deployment in k8s
environments, MinIO operator would run a
webhook service endpoint
which can be used to fetch any environment
value in a generalized approach.
2020-08-06 18:03:16 -07:00
poornasandGitHub adcaa6f9de fix: Change ListBucketTargets handler (#10217)
to list all targets across a tenant.
Also fixing some validations.
2020-08-06 17:10:21 -07:00
poornasandGitHub 121164db56 fix: relax some replication validations (#10210)
Also inherit storage class from source object
if replication configuration does not have a storage
class specified for destination bucket.
2020-08-05 20:01:20 -07:00
HarshavardhanaandGitHub a20d4568a2 fix: make sure to use uniform drive count calculation (#10208)
It is possible in situations when server was deployed
in asymmetric configuration in the past such as

```
minio server ~/fs{1...4}/disk{1...5}
```

Results in setDriveCount of 10 in older releases
but with fairly recent releases we have moved to
having server affinity which means that a set drive
count ascertained from above config will be now '4'

While the object layer make sure that we honor
`format.json` the storageClass configuration however
was by mistake was using the global value obtained
by heuristics. Which leads to prematurely using
lower parity without being requested by the an
administrator.

This PR fixes this behavior.
2020-08-05 13:31:12 -07:00
HarshavardhanaandGitHub e656beb915 feat: allow service accounts to be generated with OpenID STS (#10184)
Bonus also fix a bug where we did not purge relevant
service accounts generated by rotating credentials
appropriately, service accounts should become invalid
as soon as its corresponding parent user becomes invalid.

Since service account themselves carry parent claim always
we would never reach this problem, as the access get
rejected at IAM policy layer.
2020-08-05 13:08:40 -07:00
poornasandGitHub 88daaef76b Validate object lock when setting replication config. (#10200)
Check if object lock is enabled on
destination bucket while setting replication
configuration on a object lock enabled bucket.
2020-08-04 23:02:27 -07:00
HarshavardhanaandGitHub 0b8255529a fix: proxies set keep-alive timeouts to be system dependent (#10199)
Split the DialContext's one for internode and another
for all other external communications especially
proxy forwarders, gateway transport etc.
2020-08-04 14:55:53 -07:00
HarshavardhanaandGitHub 019fe69a57 fix: reduce an extra system call for writes instead fail later (#10187) 2020-08-04 12:09:41 -07:00
Anis ElleuchandGitHub 6ae30b21c9 fix ILM should not remove a protected version (#10189) 2020-08-03 23:04:40 -07:00
HarshavardhanaandGitHub b16781846e allow server to start even with corrupted/faulty disks (#10175) 2020-08-03 18:17:48 -07:00
HarshavardhanaandGitHub 5ce82b45da add CopyObject optimization when source and destination are same (#10170)
when source and destination are same and versioning is enabled
on the destination bucket - we do not need to re-create the entire
object once again to optimize on space utilization.

Cases this PR is not supporting

- any pre-existing legacy object will not
  be preserved in this manner, meaning a new
  dataDir will be created.

- key-rotation and storage class changes
  of course will never re-use the dataDir
2020-08-03 16:21:10 -07:00
HarshavardhanaandGitHub e99bc177c0 fix: allow FS mode situations when conflicting files exist (#10185)
conflicting files can exist on FS at
`.minio.sys/buckets/testbucket/policy.json/`, this is an
expected valid scenario for FS mode allow it to work,
i.e ignore and move forward
2020-08-03 13:20:49 -07:00
HarshavardhanaandGitHub b68bc75dad fix: quorum calculation mistake with reduced parity (#10186)
With reduced parity our write quorum should be same
as read quorum, but code was still assuming

```
readQuorum+1
```

In all situations which is not necessary.
2020-08-03 12:15:08 -07:00
HarshavardhanaandGitHub d61eac080b fix: connection_string should override other params (#10180)
closes #9965
2020-08-03 09:16:00 -07:00
poornasandGitHub a8dd7b3eda Refactor replication target management. (#10154)
Generalize replication target management so
that remote targets for a bucket can be
managed with ARNs. `mc admin bucket remote`
command will be used to manage targets.
2020-07-30 19:55:22 -07:00
HarshavardhanaandGitHub 25a55bae6f fix: avoid buffering of server sent events by proxies (#10164) 2020-07-30 19:45:12 -07:00
HarshavardhanaandGitHub fe157166ca fix: Pass context all the way down to the network call in lockers (#10161)
Context timeout might race on each other when timeouts are lower
i.e when two lock attempts happened very quickly on the same resource
and the servers were yet trying to establish quorum.

This situation can lead to locks held which wouldn't be unlocked
and subsequent lock attempts would fail.

This would require a complete server restart. A potential of this
issue happening is when server is booting up and we are trying
to hold a 'transaction.lock' in quick bursts of timeout.
2020-07-29 23:15:34 -07:00
Adam BrownandGitHub f7259adf83 Update LastUpdate timestamp before save (#10152) 2020-07-28 13:20:50 -07:00
HarshavardhanaandGitHub 6669560cb9 turn-off bucket usage metrics in gateway mode (#10150)
closes #10147
2020-07-28 13:04:26 -07:00
poornasandGitHub b46ab7e921 Rename replication target handler (#10142)
Rename replication target handler to a generic bucket target handler
2020-07-28 11:50:47 -07:00
HarshavardhanaandGitHub 27266f8a54 fix: if OPA set do not enforce policy claim (#10149) 2020-07-28 11:47:57 -07:00