Error code 101 on a systemd-triggered service

kelvin · October 24, 2020, 4:48am

Hi!

I had a VM (Proxmox, LXC-based container) running a bitwarden_rs 1.12.0 on an Ubuntu 18.04; have just got it dist-upgraded to Ubuntu 20.04 and recompiled 1.17.0 on it. However, once I try to start the service just like usual, I get an error code 101. That error would not appear when I just bitwarden_rs from the command line (with all the same environment variables from the env file), so I am not sure if that is really a bug or misconfiguration on my side.

I can reproduce the same behaviour by copying the same set of env, service file, compiled executable and web vault files to a fresh Ubuntu 20.04 VM instead of just the original dist-upgraded VM. The same behaviour also occurs in both the debug and the release build.

Using the debug build, the following is got from the journalctl -xe -u bitwarden_rs.service:

Oct 24 04:18:21 temp systemd[1]: Started Bitwarden Server (Rust Edition).
-- Subject: A start job for unit bitwarden_rs.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit bitwarden_rs.service has finished successfully.
-- 
-- The job identifier is 618.
Oct 24 04:18:22 temp bitwarden_rs[3042]: /--------------------------------------------------------------------\
Oct 24 04:18:22 temp bitwarden_rs[3042]: |                       Starting Bitwarden_RS                        |
Oct 24 04:18:22 temp bitwarden_rs[3042]: |                           Version 1.17.0                           |
Oct 24 04:18:22 temp bitwarden_rs[3042]: |--------------------------------------------------------------------|
Oct 24 04:18:22 temp bitwarden_rs[3042]: | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
Oct 24 04:18:22 temp bitwarden_rs[3042]: | official channels to report bugs/features, regardless of client.   |
Oct 24 04:18:22 temp bitwarden_rs[3042]: | Send usage/configuration questions or feature requests to:         |
Oct 24 04:18:22 temp bitwarden_rs[3042]: |   https://vaultwarden.discourse.group/                             |
Oct 24 04:18:22 temp bitwarden_rs[3042]: | Report suspected bugs/issues in the software itself at:            |
Oct 24 04:18:22 temp bitwarden_rs[3042]: |   https://github.com/dani-garcia/bitwarden_rs/issues/new           |
Oct 24 04:18:22 temp bitwarden_rs[3042]: \--------------------------------------------------------------------/
Oct 24 04:18:24 temp bitwarden_rs[3042]: [2020-10-24 04:18:24.430][panic][ERROR] thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }': /root/.cargo/registry/src/github.com-1ecc6299db9ec823/scheduled-thread-pool-0.2.5/src/lib.rs:320
Oct 24 04:18:24 temp bitwarden_rs[3042]:    0: bitwarden_rs::init_logging::{{closure}}
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/src/bitwarden_rs/src/main.rs:158:25
Oct 24 04:18:24 temp bitwarden_rs[3042]:    1: std::panicking::rust_panic_with_hook
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/panicking.rs:530:17
Oct 24 04:18:24 temp bitwarden_rs[3042]:    2: rust_begin_unwind
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/panicking.rs:437:5
Oct 24 04:18:24 temp bitwarden_rs[3042]:    3: core::panicking::panic_fmt
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libcore/panicking.rs:85:14
Oct 24 04:18:24 temp bitwarden_rs[3042]:    4: core::option::expect_none_failed
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libcore/option.rs:1269:5
Oct 24 04:18:24 temp bitwarden_rs[3042]:    5: core::result::Result<T,E>::unwrap
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libcore/result.rs:1005:23
Oct 24 04:18:24 temp bitwarden_rs[3042]:    6: scheduled_thread_pool::Worker::start
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/scheduled-thread-pool-0.2.5/src/lib.rs:320:9
Oct 24 04:18:24 temp bitwarden_rs[3042]:    7: scheduled_thread_pool::ScheduledThreadPool::new_inner
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/scheduled-thread-pool-0.2.5/src/lib.rs:157:13
Oct 24 04:18:24 temp bitwarden_rs[3042]:    8: scheduled_thread_pool::ScheduledThreadPool::with_name
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/scheduled-thread-pool-0.2.5/src/lib.rs:136:9
Oct 24 04:18:24 temp bitwarden_rs[3042]:    9: r2d2::config::Builder<M>::build_unchecked
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/r2d2-0.8.9/src/config.rs:250:30
Oct 24 04:18:24 temp bitwarden_rs[3042]:   10: r2d2::config::Builder<M>::build
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/r2d2-0.8.9/src/config.rs:227:20
Oct 24 04:18:24 temp bitwarden_rs[3042]:   11: bitwarden_rs::db::DbPool::from_config
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/src/bitwarden_rs/src/db/mod.rs:54:40
Oct 24 04:18:24 temp bitwarden_rs[3042]:   12: core::ops::function::Fn::call
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libcore/ops/function.rs:72:5
Oct 24 04:18:24 temp bitwarden_rs[3042]:   13: bitwarden_rs::util::retry_db
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/src/bitwarden_rs/src/util.rs:445:15
Oct 24 04:18:24 temp bitwarden_rs[3042]:   14: bitwarden_rs::launch_rocket
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/src/bitwarden_rs/src/main.rs:271:22
Oct 24 04:18:24 temp bitwarden_rs[3042]:   15: bitwarden_rs::main
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /root/src/bitwarden_rs/src/main.rs:70:5
Oct 24 04:18:24 temp bitwarden_rs[3042]:   16: std::rt::lang_start::{{closure}}
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/rt.rs:67:34
Oct 24 04:18:24 temp bitwarden_rs[3042]:   17: std::rt::lang_start_internal::{{closure}}
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/rt.rs:52:13
Oct 24 04:18:24 temp bitwarden_rs[3042]:       std::panicking::try::do_call
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/panicking.rs:348:40
Oct 24 04:18:24 temp bitwarden_rs[3042]:       std::panicking::try
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/panicking.rs:325:15
Oct 24 04:18:24 temp bitwarden_rs[3042]:       std::panic::catch_unwind
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/panic.rs:394:14
Oct 24 04:18:24 temp bitwarden_rs[3042]:       std::rt::lang_start_internal
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/rt.rs:51:25
Oct 24 04:18:24 temp bitwarden_rs[3042]:   18: std::rt::lang_start
Oct 24 04:18:24 temp bitwarden_rs[3042]:              at /rustc/daecab3a784f28082df90cebb204998051f3557d/src/libstd/rt.rs:67:5
Oct 24 04:18:24 temp bitwarden_rs[3042]:   19: main
Oct 24 04:18:24 temp bitwarden_rs[3042]:   20: __libc_start_main
Oct 24 04:18:24 temp bitwarden_rs[3042]:   21: _start
Oct 24 04:18:24 temp systemd[1]: bitwarden_rs.service: Main process exited, code=exited, status=101/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit bitwarden_rs.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 101.
Oct 24 04:18:24 temp systemd[1]: bitwarden_rs.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit bitwarden_rs.service has entered the 'failed' state with result 'exit-code'.

/etc/systemd/system/bitwarden_rs.service:

[Unit]
Description=Bitwarden Server (Rust Edition)
Documentation=https://github.com/dani-garcia/bitwarden_rs
After=network.target

[Service]
User=root
Group=root
EnvironmentFile=/etc/bitwarden_rs.env
Type=simple
ExecStart=/usr/bin/bitwarden_rs
ExecStop=/usr/bin/killall bitwarden_rs
#Restart=always
#RestartSec=30s
LimitNOFILE=1024
LimitNPROC=4
#PrivateTmp=true
#PrivateDevices=true
#ProtectHome=true
#NoNewPrivileges=true
#ProtectSystem=strict
WorkingDirectory=/var/lib/bitwarden_rs
ReadWriteDirectories=/var/lib/bitwarden_rs /var/log/bitwarden_rs
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

/etc/bitwarden_rs.env:

WEB_VAULT_FOLDER=/var/lib/bitwarden_rs/webbuild/
WEB_VAULT_ENABLED=true
WEBSOCKET_ENABLED=true
WEBSOCKET_ADDRESS=0.0.0.0
WEBSOCKET_PORT=3012
LOG_LEVEL=info
SIGNUPS_ALLOWED=false
ROCKET_ADDRESS=0.0.0.0
ROCKET_PORT=80
ROCKET_WORKERS=1

If that helps, I can upload strace for both service bitwarden_rs start and bitwarden_rs log for your reference too. Thank you.

BlackDex · November 23, 2020, 5:00pm

Looks like something went wrong during the building of bitwarden_rs.
Could it be that you have used a different version for the rust-toolchain?

The one used by default is nightly-2020-10-14.

kelvin · November 24, 2020, 5:06pm

Thanks @BlackDex for your reply!

Indeed - I have installed mine with curl https://sh.rustup.rs -sSf | sh before and I got the stable version rustc 1.47.0 (18bf6b4f0 2020-10-07).

However, since I am using the source at 1.17.0 branch, thus instead of nightly-2020-10-14 I have changed to nightly-2020-07-11 and recompiled:

rustup toolchain install nightly-2020-07-11
rustup default nightly-2020-07-11
cargo clean
cargo build -j 2 --features sqlite --release

I actually have tried both release and debug build and both seem to have the similar problem. Triggering with systemd returns error but starting standalone is OK.

Now that I have tried again, I think it might be more likely this problem lies in Proxmox/Systemd side rather than Bitwarden_rs…?

benpro · January 31, 2021, 12:07am

On LXC you need to disable LimitNPROC=64 in the systemd unit file.

BlackDex · January 31, 2021, 9:56pm

@benpro is right. Also see: bitwarden_rs crashes from some kind of panic · Issue #1202 · dani-garcia/bitwarden_rs · GitHub

And: Setup as a systemd service · dani-garcia/bitwarden_rs Wiki · GitHub

kelvin · February 2, 2021, 6:40pm

Thank you @benpro and @BlackDex! I have confirmed on my Ubuntu 20.04 instance (on Proxmox 6.3, with systemd 245) that removing the LimitNPROC=64 does make it start successfully. It would also work with the Private* and Protect* parameters enabled as in the service file above.

Thanks again!

Topic		Replies	Views
Systemd-docker timout on shutdown Third Party Help	1	546	September 11, 2020
The error for starting the container of bitwardenrs Help	2	2202	January 15, 2021
Problem starting bitwarden_rs Help	0	599	March 1, 2021
Firewall rules, apache proxy and docker Third Party Help	2	1525	October 6, 2020
Change domain name? Help	5	3248	February 8, 2021

Error code 101 on a systemd-triggered service

Related Topics