Request limits #33

2026-10-29T15:07:09Z

fauno commented

2026-10-29 15:07:09 +00:00

@Brooke i'm considering setting rate limits following block_max_concurrent_reads and block_max_concurrent_writes_per_request because we've been seeing a lot of 504 gateway timeout errors on relatively high load (higher than normal anyway), but setting it to 16 was too low and now we have a lot of 429 errors. i'm not sure what the average should be yet and if they're related to the two options i mentioned. also i don't know if the concurrent reads are per hdd or per node.

@Brooke i'm considering setting rate limits following [block_max_concurrent_reads](https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#block_max_concurrent_reads) and [block_max_concurrent_writes_per_request](https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#block_max_concurrent_writes_per_request) because we've been seeing a lot of 504 gateway timeout errors on relatively high load (higher than normal anyway), but setting it to 16 was too low and now we have a lot of 429 errors. i'm not sure what the average should be yet and if they're related to the two options i mentioned. also i don't know if the concurrent reads are per hdd or per node.

fauno added 2 commits 2026-10-29 15:07:10 +00:00

fix: make sure we don't set an upload limit 932438847e

doc: note about upload limits

continuous-integration/drone/pr Build is failing

Details

8910018d47

fauno requested review from Brooke 2026-10-29 15:07:10 +00:00

fauno commented

2026-10-29 15:09:38 +00:00

if garage follows the s3 spec for multipart uploads, the max body size should be 5GiB

if garage follows the s3 spec for multipart uploads, the [max body size should be 5GiB](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html)

Brooke commented

2026-10-29 16:43:34 +00:00

adding them as options to .env sounds like a good idea. I generally feel like it is ill advised to set an unlimited max body size. Under the known issue page they seem to address this by saying there isn't currently a max? Unless I'm reading this wrong should we also set a standard block size? https://garagehq.deuxfleurs.fr/documentation/reference-manual/known-issues/

fauno commented

2026-10-29 19:42:23 +00:00

do you mean this? https://garagehq.deuxfleurs.fr/documentation/reference-manual/known-issues/#very-big-objects-cause-performance-degradation

i read it as that they support any stored object size but the upload should be multipart. according to s3 docs anything bigger than 100MB should be uploaded as multipart, but chunks can range from 5MB to 5GB.

do you mean this? https://garagehq.deuxfleurs.fr/documentation/reference-manual/known-issues/#very-big-objects-cause-performance-degradation i read it as that they support any stored object size but the upload should be multipart. according to s3 docs anything bigger than 100MB should be uploaded as multipart, but chunks can range from 5MB to 5GB.

👍 1

fauno added 1 commit 2026-10-29 20:09:27 +00:00

feat: limit uploaded chunk size

continuous-integration/drone/pr Build is failing

Details

a826717ef3

fauno commented

2026-10-29 20:11:06 +00:00

ok i looked around different clients and the garage source code and i think 5GiB as an upper limit is enough, and it could also start at 5MiB but it depends on users' needs.

fauno added 1 commit 2026-10-29 20:12:11 +00:00

fix: typo

continuous-integration/drone/pr Build is failing

Details

be00ad1823

fauno commented

2026-10-29 20:51:41 +00:00

just for documenting this somewhere. i'm not sure it can be achieved with traefik rate limits, and the 3 concurrent writes per node seems two low for a gateway, maybe it can be concurrent writes * nodes, and the same for reads.

this comes from observing a some 504 gateway errors while doing backups, which sometimes the backup software doesn't understand to retry, and a friend doing a proper load test where errors were measured. garage is the source of the 504 error. come to think of it i should first investigate in which cases it returns this error.

since our gateways are nginx not deployed by coopcloud, i was trying to set rate limits but apparently it's impossible to set a rate limit dynamically by request_method without a commercial subscription.

this would work with a commercial subscription:

map $request_method $rate_by_request_method {
  default 16r/s;
  HEAD 1000r/s;
  PUT 3r/s;
}

limit_req_zone $server_name zone=garage_req:10m rate=$rate_by_request_method;

server {
  limit_req zone=garage_req;
}

this doesn't work because the zone parameter doesn't expand variables:

map $request_method $req_zone_by_request_method {
  default garage_req_16;
  HEAD garage_req_1000;
  PUT garage_req_3;
}

limit_req_zone $server_name zone=garage_req_1000:10m rate=1000r/s;
limit_req_zone $server_name zone=garage_req_16:10m rate=16r/s;
limit_req_zone $server_name zone=garage_req_3:10m rate=3r/s;

server {
  limit_req zone=$req_zone_by_request_method;
}

this doesn't work because limit_req is not allowed within an if context:

map $request_method $req_zone_by_request_method {
  default garage_req_16;
  HEAD garage_req_1000;
  PUT garage_req_3;
}

limit_req_zone $server_name zone=garage_req_16:10m rate=16r/s;
limit_req_zone $server_name zone=garage_req_3:10m rate=3r/s;

server {
  if ($request_method = 'GET') {
    limit_req zone=garage_req_16;
  }

  if ($request_method = 'PUT') {
    limit_req zone=garage_req_3;
  }

}

just for documenting this somewhere. i'm not sure it can be achieved with traefik rate limits, and the 3 concurrent writes per node seems two low for a gateway, maybe it can be concurrent writes * nodes, and the same for reads. this comes from observing a some 504 gateway errors while doing backups, which sometimes the backup software doesn't understand to retry, and a friend doing a proper load test where errors were measured. garage is the source of the 504 error. come to think of it i should first investigate in which cases it returns this error. since our gateways are nginx not deployed by coopcloud, i was trying to set rate limits but apparently it's impossible to set a rate limit dynamically by `request_method` without a commercial subscription. this would work with a commercial subscription: ```nginx map $request_method $rate_by_request_method { default 16r/s; HEAD 1000r/s; PUT 3r/s; } limit_req_zone $server_name zone=garage_req:10m rate=$rate_by_request_method; server { limit_req zone=garage_req; } ``` this doesn't work because the zone parameter doesn't expand variables: ```nginx map $request_method $req_zone_by_request_method { default garage_req_16; HEAD garage_req_1000; PUT garage_req_3; } limit_req_zone $server_name zone=garage_req_1000:10m rate=1000r/s; limit_req_zone $server_name zone=garage_req_16:10m rate=16r/s; limit_req_zone $server_name zone=garage_req_3:10m rate=3r/s; server { limit_req zone=$req_zone_by_request_method; } ``` this doesn't work because `limit_req` is not allowed within an `if` context: ```nginx map $request_method $req_zone_by_request_method { default garage_req_16; HEAD garage_req_1000; PUT garage_req_3; } limit_req_zone $server_name zone=garage_req_16:10m rate=16r/s; limit_req_zone $server_name zone=garage_req_3:10m rate=3r/s; server { if ($request_method = 'GET') { limit_req zone=garage_req_16; } if ($request_method = 'PUT') { limit_req zone=garage_req_3; } } ```

Brooke commented

2026-10-30 04:12:26 +00:00

ok i looked around different clients and the garage source code and i think 5GiB as an upper limit is enough, and it could also start at 5MiB but it depends on users' needs.

This makes sense, I was reading that known issues page incorrectly

> ok i looked around different clients and the garage source code and i think 5GiB as an upper limit is enough, and it could also start at 5MiB but it depends on users' needs. This makes sense, I was reading that known issues page incorrectly

Brooke approved these changes

2026-11-07 13:02:43 +00:00

Brooke left a comment

If this pull is limited to the traefik middleware it seems fine, but I would put the upload limits before the "for more" line in README.md as it's meant to be the last line. Not blocking.

fauno added 1 commit 2026-11-08 14:35:49 +00:00

fix: change order

continuous-integration/drone/pr Build is failing

Details

c712e9a47a

fauno merged commit e349388f20 into main

2026-11-08 14:36:10 +00:00

fauno deleted branch request-limits

2026-11-08 14:36:10 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/garage#33