"Disk Quota Exceeded" when trying to add container

[SirHades]

The system is a TS-873A with a 24TB raid zfs array, 512GB NVMe raid System array and 32Gb RAM.

There's enough space left on every share, at least 75% RAM available and quotas are disabled. This is a local machine, for our household. No ports are redirected from the outside world. There's users for me, my wife, docker images and the raspberry pi which handles PiHole and Wireguard. None of them have quotas.

I have Container Station running since I installed this NAS. Until yesterday, there were 6 containers installed, including Portainer, which I use to install / update other containers. I always remove unused images to avoid confusion / clutter.

Yesterday I removed Nextcloud container from it and wanted to recreate it from scratch. As I tried to install it in Portainer, it simply went back to the "add container page". I must have missed the error message. Did the same in Container Station, and this is when I first got the error "disk quota exceeded".

The full error message is almost the same with every try, except for the file it is trying to write when the quota is reached/exceeded.

" Background task error for create #DockerName#: failed to register layer: ApplyLayer exit status 1 stdout: stderr: write /#whichever file it's trying to write#: disk quota exceeded "

I then restarted container station, and it installed normally.

Today I tried to install LibreOffice, and had the same error. Restarted Container station, but it's still not working.

I also tried in cli, and simply pulling the image, but it gives the same error, no luck there to get more info. It downloads it all, but gives the error while halfway through extraction.

I also noticed yesterday (since I make most of my docker stuff in portainer) that container station indicates over 80% memory usage. docker stats gives me about 1.5% mem usage, or about 400-450Mib. The amount of memory used according to Container Station doesn't make any sense.

Last thing to try was restarting the nas. Not ideal, but it now shows 42% of memory used in Container Station, and installed the new container without any complaints.
Later today I tried to add another container, see if the restart was a miracle, but alas, even restarting it a second time didn't work this time. This leaves me with no way to update my containers.

Why do I get a disk quota error when there's no quota configured?
And as a side note, why does Container Station give me this huge memory usage?

Cheers for your help,

.H

[tweexter]

Regarding the memory issues, check out this other post - viewtopic.php?p=806011#p806011

Unfortunately I'm having the same issue with disk space error on container station / portainer, but I definitely have no such space restraints with over 30TB.

Hoping someone has some ideas here...

[fabriziorizzo]

Anybody figure this out yet?

I'm in the same boat. SOME of my containers consistently have this issue now, many continue to pull and function normally. The packages that are not pulling properly have pulled and worked just fine in the past. QNAP's borked something somewhere... as usual. No idea what though.

I tried removing container station completely, cleaning up all .qpkg/container-station[-data] folders on all volumes, removing my docker user and re-creating, verifying no quota is set, rebooting, re-installing Container Station, and trying to pull everything fresh - both from CLI as shown here, and from within the Container Station GUI.

From what I can gather from vague details found on-line, this sort of issue possibly has to do with something not set correctly with the docker install's "rootless" config.
Considering all of these containers were working for a couple of years before a periodic pull to snag updates started failing... It's very likely either a change in one of the QNAP firmware versions, or in a Container Station version update within the last 9-months or so.

Also posted to the LSIO container discourse, trying to get advise from a container-specific forum.
https://discourse.linuxserver.io/t/ombi ... ml/3953/15
For the "omni" container, one of the folks there was able to deploy to his QNAP that's running v5.x firmware... so maybe this is something that's been fixed but not back ported to firmware v4.5.4 or v2.4.3+ container station...


I'm on:
QNAP firmware v 4.5.4.1931 (2022-01-28)
Container Station v.2.4.3.208 (2022-02-13)

Code: Select all

# docker version
Client:
 Version:           20.10.7-qnap3
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        50b64c4
 Built:             Tue Oct 26 07:03:45 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7-qnap3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f180ce8
  Built:            Tue Oct 26 07:05:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Code: Select all

[/share/docker/mediasvcs] # docker-compose pull                         
Pulling authelia      ... done
Pulling swag          ... done
Pulling sabnzbd       ... done
Pulling grafana       ... done
Pulling radarr        ... extracting (100.0%)
Pulling sonarr        ... done
Pulling lidarr        ... extracting (100.0%)
Pulling lazylibrarian ... done
Pulling ombi          ... extracting (100.0%)
Pulling heimdall      ... done

ERROR: for radarr  failed to register layer: Error processing tar file(exit status 1): lchown /app/radarr/bin/System.Private.DataContractSerialization.dll: disk quota exceeded

ERROR: for ombi  failed to register layer: Error processing tar file(exit status 1): lchown /opt/ombi/Ombi: disk quota exceeded

ERROR: for lidarr  failed to register layer: Error processing tar file(exit status 1): lchown /app/lidarr/bin/BouncyCastle.Crypto.dll: disk quota exceeded
ERROR: failed to register layer: Error processing tar file(exit status 1): lchown /app/lidarr/bin/BouncyCastle.Crypto.dll: disk quota exceeded

[fabriziorizzo]

I was able to workaround the issue (without any useful aid from Support).

Initial state:
Control-Panel > Privileges > Users showed "-" for the Quota column, was set to "no limit" for all users's profiles.
NOTE: this tells me that quotas were actually on and the UI was not representing things correctly. If quotas were actually disabled, the Edit User Profile view for a given user would show Quota: disabled (in red text). It did not.

Control-Panel > Privileges > Quota showed Enable quotas for all users as [UNCHECKED], and quota size on disk: [empty field], with no table of users, etc... shown - clearly indicating that quotas were turned off, even though the user profile view did not show "Quota: disabled" for each user.

Workaround:
1. enabled quotas for all users, applying the following setting:
Control-Panel > Privileges > Quota: [CHECKED] Enable quotas for all users, Quota size on disk: 2048GB

2. waiting and refreshing the control panel until quotas showed up on that screen

3. Individually disabled quotas for each user profile one-at-a-time
Control-Panel > Privileges > Users: click on the "edit account profile" glyph for each user row, set to no limit.

4. Turn quotas off for all users
Control-Panel > Privileges > Quota: [UNCHECKED] Enable quotas for all users

Validation:
5. Check individual user profile quota details to verify "Quota: disabled" instead of "Quota: (selected) no limit"
6. Execute successfully: docker pull linuxserver/radarr; docker pull linuxserver/lidarr; docker pull linuxserver/ombi

Root-Cause Analysis:
Quotas were evidently on and in an inconsistent state even though:
Control-Panel > Privileges > Quota showed "Enable quotas for all users" as UNCHECKED.
User Profile details:
Control-Panel > Privileges > Users: click on the "edit account profile" glyph for each user row;
displayed the quota section with 3 selectable options and the "no limit" option selected instead of simply "Quota: disabled"

Whatever state this left the actual filesystems in with regards to quotas was sufficient to cause problems when the docker engine attepted to execute a chown on files in a docker volume overlay. The issue, however was not sufficient to cause problems with pulling images and writing them to disk in the first place.

Why the partially and inconsistently configured quotas caused problems with chown by a named non-root user within the context of a docker volume overlay, but not via command shell on the NAS directly remains a mystery.

[toomuchstuff]

This resolved the issue for me. Thanks so much for your time investigating and sharing. Not sure if QNAP support every got back to you.

[fabriziorizzo]

Thanks. No, they did not…

[mseewald]

Thanks. Hugely helpful!

[vger74656]

Thanks for this, worked for me too.