Commit Graph
5711 Commits
Author SHA1 Message Date
jiukerandGitHub 2cd98a0d21 remove outdated notes (#18755) 2024-01-08 08:04:19 -08:00
Anis EleuchandGitHub 04135fa6cd audit: Add the drives where the dangling object is removed (#18737) 2024-01-05 14:17:24 -08:00
HarshavardhanaandGitHub 42dc6329e6 simplify success response for GetObjectAttributes() (#18746) 2024-01-05 12:50:07 -08:00
SveinnandGitHub 9b8ba97f9f feat: add support for GetObjectAttributes API (#18732) 2024-01-05 10:43:06 -08:00
Anis EleuchandGitHub 7705605b5a scanner: Add a config to disable short sleep between objects scan (#18734)
Add a hidden configuration under the scanner sub section to configure if
the scanner should sleep between two objects scan. The configuration has
only effect when there is no drive activity related to s3 requests or
healing.

By default, the code will keep the current behavior which is doing
sleep between objects.

To forcefully enable the full scan speed in idle mode, you can do this:

   `mc admin config set myminio scanner idle_speed=full`
2024-01-04 15:07:17 -08:00
Anis EleuchandGitHub 414bcb0c73 prom: Add read quorum per erasure set metric (#18736) 2024-01-04 15:05:13 -08:00
HarshavardhanaandGitHub f4710948c4 fix: an odd crash when deleting null DEL markers (#18727)
fixes #18724

A regression was introduced in #18547, that attempted
to file adding a missing `null` marker however we
should not skip returning based on versionID instead
it must be based on if we are being asked to create
a DEL marker or not.

The PR also has a side-affect for replicating `null`
marker permanent delete, as it may end up adding a
`null` marker while removing one.

This PR should address both scenarios.
2024-01-02 15:08:18 -08:00
Anis EleuchandGitHub 3f4488c589 scanner: Allow full throttle if there is no parallel disk ops (#18109) 2024-01-02 13:51:24 -08:00
Pedro JuarezandGitHub 8f13c8c3bf Support to store browser config settings (#18631)
* csp_policy
* hsts_seconds
* hsts_include_subdomains
* hsts_preload
* referrer_policy
2024-01-01 08:36:33 -08:00
Zhou TingandGitHub 31d16f6cc2 allow sha256 payload to be configurable for object perf test (#18712)
Signed-off-by: Zhou Ting <ting.z.zhou@intel.com>
2023-12-29 23:56:50 -08:00
HarshavardhanaandGitHub a50ea92c64 feat: introduce list_quorum="auto" to prefer quorum drives (#18084)
NOTE: This feature is not retro-active; it will not cater to previous transactions
on existing setups. 

To enable this feature, please set ` _MINIO_DRIVE_QUORUM=on` environment
variable as part of systemd service or k8s configmap. 

Once this has been enabled, you need to also set `list_quorum`. 

```
~ mc admin config set alias/ api list_quorum=auto` 
```

A new debugging tool is available to check for any missing counters.
2023-12-29 15:52:41 -08:00
HarshavardhanaandGitHub 5b2ced0119 re-use globalLocalDrives properly (#18721) 2023-12-29 09:30:10 -08:00
Anis EleuchandGitHub 8a0ba093dd audit: Fix merrs and derrs object dangling message (#18714)
merrs and derrs are empty when a dangling object is deleted. Fix the bug
and adds invalid-meta data for data blocks
2023-12-27 22:27:04 -08:00
Daniel ValdiviaandGitHub 5fc7da345d Upgrade Console to v0.44.0 (#18717)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-12-27 11:19:13 -08:00
Anis EleuchandGitHub 8bd4f6568b server-info: Avoid initializing audit/log http/kafka targets (#18703)
This can cause unnecessary ServerInfo() call delay.
2023-12-22 10:25:08 -08:00
HarshavardhanaandGitHub da55499db0 fix: reject clients that do not send proper payload (#18701) 2023-12-22 01:26:17 -08:00
Anis EleuchandGitHub 22f8e39b58 tier: Allow edit of the new Azure and AWS auth params (#18690)
Allow editing for the service principal credentials from Azure
and the web identity token for AWS;

Also, more validation of input parameters.
2023-12-21 16:58:10 -08:00
HarshavardhanaandGitHub eba23bbac4 rename object_size -> block_size for cache subsystem (#18694) 2023-12-21 16:57:13 -08:00
HarshavardhanaandGitHub 4550535cbb send proper IPv6 names avoid bracketing notation (#18699)
Following policies if present

```
       "Condition": {
         "IpAddress": {
            "aws:SourceIp": [
              "54.240.143.0/24",
               "2001:DB8:1234:5678::/64"
             ]
          }
        }
```

And client is making a request to MinIO via IPv6 can
potentially crash the server.

Workarounds are turn-off IPv6 and use only IPv4
2023-12-21 16:56:55 -08:00
Anis EleuchandGitHub 8432fd5ac2 prom: Add online and healing drives metrics per erasure set (#18700) 2023-12-21 16:56:43 -08:00
HarshavardhanaandGitHub 7c948adf88 allow pre-allocating buffers to reduce frequent GCs during growth (#18686)
This PR also increases per node bpool memory from 1024 entries
to 2048 entries; along with that, it also moves the byte pool
centrally instead of being per pool.
2023-12-21 08:59:38 -08:00
Krishnan ParthasarathiandGitHub 56b7045c20 Export tier metrics (#18678)
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
2023-12-20 20:13:40 -08:00
PoornaandGitHub d55b6b9909 Fix quota config replication for SR (#18684)
Fixing regression introduced by PR #17988
2023-12-19 13:22:47 -08:00
Shireesh AnjalandGitHub 7680e5f81d Read new key license_v2 from SUBNET response (#18669)
SUBNET now has a v2 of license that is returned in the new key
`license_v2`. mc will start reading and storing the same. (The old key
`license` is deprecated but is still available in SUBNET response to
ensure that the current released version of minio doesn't break)
2023-12-18 08:21:44 -08:00
Taran PelkeyandGitHub ad8a34858f Add APIs to create and list access keys for LDAP (#18402) 2023-12-15 13:00:43 -08:00
Krishnan ParthasarathiandGitHub 162eced7d2 Fix incorrect metric desc for bucketRequestsDuration (#18657) 2023-12-14 19:02:11 -08:00
Krishnan ParthasarathiandGitHub bec1f7c26a metrics: Refactor handling of histogram vectors (#18632) 2023-12-14 14:02:52 -08:00
Anis EleuchandGitHub 8771617199 tier: Add support of AWS S3 tiering with web identity token file (#18648) 2023-12-14 14:01:49 -08:00
Klaus PostandGitHub 6c89a81af4 Fix CreateFile shared buffer corruption. (#18652)
`(*xlStorageDiskIDCheck).CreateFile` wraps the incoming reader in `xioutil.NewDeadlineReader`.

The wrapped reader is handed to `(*xlStorage).CreateFile`. This performs a Read call via `writeAllDirect`, 
which reads into an `ODirectPool` buffer.

`(*DeadlineReader).Read` spawns an async read into the buffer. If a timeout is hit while reading, 
the read operation returns to `writeAllDirect`. The operation returns an error and the buffer is reused.

However, if the async `Read` call unblocks, it will write to the now recycled buffer.

Fix: Remove the `DeadlineReader` - it is inherently unsafe. Instead, rely on the network timeouts. 
This is not a disk timeout, anyway.

Regression in https://github.com/minio/minio/pull/17745
2023-12-14 10:51:57 -08:00
Praveen raj ManiandGitHub 10ca0a6936 Label the notification target metrics by their target IDs (#18633)
This patch adds the targetID to the existing notification target metrics
and deprecates the current target metrics which points to the overall
event notification subsystem
2023-12-14 09:09:26 -08:00
HarshavardhanaandGitHub b3314e97a6 re-use the same local drive used by remote-peer (#18645)
historically, we have always kept storage-rest-server
and a local storage API separate without much trouble,
since they both can independently operate due to no
special state() between them.

however, over some time, we have added state()
such as

- drive monitoring threads now there will be "2" of
  them per drive instead of just 1.

- concurrent tokens available per drive are now twice
  instead of just single shared, allowing unexpectedly
  high amount of I/O to go through.

- applying serialization by using walkMutexes can now
  be adequately honored for both remote callers and local
  callers.
2023-12-13 19:27:55 -08:00
PoornaandGitHub 3781a0f9ad replication: Pass metadata timestamps in CopyObject call (#18647)
Regression from #18285. CopyObject options were inheriting source MTime
for metadata timestamps if unspecified, removing this prevented metadata
updates from being applied on target.
2023-12-13 15:28:55 -08:00
PoornaandGitHub e79b289325 fix datadir missing check on HeadObject (#18646)
versions pending purge in replication were seeing a errFileCorrupt
that prevents permanent deletion after replication.

Regression from PR#18477
2023-12-13 14:54:01 -08:00
HarshavardhanaandGitHub 3f72c7fcc7 healthcheck requests with user-agent mozilla do not need redirects (#18642)
apparently, windows powershell curl has this abhorrent behavior
2023-12-12 16:16:26 -08:00
HarshavardhanaandGitHub d521c84d55 reduce logging during permission denied errors (#18641)
log them if any only once
2023-12-12 16:11:17 -08:00
4a21dce2b5 tier: Add support of SP credentials with Azure (#18630)
Co-authored-by: Anis Elleuch <anis@min.io>
2023-12-11 21:51:53 -08:00
HarshavardhanaandGitHub 65f34cd823 fix: remove ODirectReader entirely since we do not need it anymore (#18619) 2023-12-09 10:17:51 -08:00
HarshavardhanaandGitHub 196e7e072b allow bitrot files to be healed in MRF (#18618)
bitrot scanMode was ignored in MRF,
allow it to heal relevant content if
needed when seen as an error.
2023-12-08 12:26:01 -08:00
Anis EleuchandGitHub 6f97663174 yml-config: Add support of rootUser and rootPassword (#18615)
Users can define the root user and password in the yaml configuration
file; Root credentials defined in the environment variable still take
precedence
2023-12-08 12:04:54 -08:00
Anis EleuchandGitHub aed7a1818a info: Populate pool/set/disk indexes for offline disks (#18613)
This can be calculated from the disk layout and some external
applications would like to know the location of the offline
disks.
2023-12-08 08:13:04 -08:00
PoornaandGitHub 6b06da76cb add configuration to limit replication workers (#18601) 2023-12-07 16:22:00 -08:00
jiukerandGitHub 6ca6788bb7 feat: add events_errors_total metric (#18610) 2023-12-07 16:21:17 -08:00
Anis EleuchandGitHub 2e23e61a45 Add support of conf file to pass arguments and options (#18592) 2023-12-07 01:33:56 -08:00
HarshavardhanaandGitHub 53ce92b9ca fix: use the right channel to feed the data in (#18605)
this PR fixes a regression in batch replication
where we weren't sending any data from the Walk()
results due to incorrect channels being used.
2023-12-06 18:17:03 -08:00
Shireesh AnjalandGitHub 7350a29fec Capture percentage of cpu load and memory used (#18596)
By default the cpu load is the cumulative of all cores. Capture the
percentage load (load * 100 / cpu-count)

Also capture the percentage memory used (used * 100 / total)
2023-12-06 13:19:59 -08:00
jiukerandGitHub 5cc2c62c66 fix: GetFreePort() will get the same port (#18604) 2023-12-06 10:36:42 -08:00
HarshavardhanaandGitHub 4bc5ed6c76 support LDAP service accounts via SFTP, FTP logins (#18599) 2023-12-06 04:31:35 -08:00
HarshavardhanaandGitHub 73dde66dbe stick to go1.19 go.mod (#18600) 2023-12-06 01:09:22 -08:00
Harshavardhana e30c0e7ca3 Revert "Heal buckets at node level (#18504)"
This reverts commit 708296ae1b.
2023-12-05 22:34:46 -08:00
ShubhenduandGitHub 708296ae1b Heal buckets at node level (#18504) 2023-12-05 02:17:35 -08:00