Your systemd Service Will Not Start and the Logs Say Nothing Useful

The short answer

Job for myapp.service failed because the control process exited with error code.
See "systemctl status myapp.service" and "journalctl -xeu myapp.service" for details.

That message contains no information. These three commands do:

# the exit status, which is the actual clue
systemctl status myapp.service --no-pager -l

# logs for this unit only, this boot, with explanations
journalctl -xeu myapp.service -b --no-pager

# what systemd thinks it is running, after merging drop-ins
systemctl cat myapp.service
systemctl show myapp.service -p ExecStart -p User -p WorkingDirectory -p Environment

The number in status= is the whole diagnosis. 203/EXEC means the binary was not executable or not found. 217/USER means the user does not exist. 1 means your program ran and chose to exit.

Tested on systemd 255, Ubuntu 24.04, Linux 6.8.

Reading the status codes

Systemd reserves a range of exit codes for failures that happened before your program ever ran. Anything in this table means the fault is in the unit file, not your application.

The distinction matters because it tells you which half of the system to look at. Everything above 200 happened in systemd. Plain 1 happened in your code.

The causes, in the order I check them

1. 203/EXEC and the shell you do not have

ExecStart is not a shell command. Systemd executes the binary directly, so none of this works:

# all four of these fail
ExecStart=npm start
ExecStart=node server.js && echo started
ExecStart=/usr/bin/node server.js > /var/log/app.log
ExecStart=NODE_ENV=production /usr/bin/node server.js

No PATH lookup, no operators, no redirection, no inline environment. Give it an absolute path:

ExecStart=/usr/bin/node /srv/app/server.js
Environment=NODE_ENV=production
StandardOutput=journal

If you genuinely need shell features, invoke a shell on purpose:

ExecStart=/bin/bash -lc '/srv/app/bin/start.sh'

For a script, 203/EXEC also fires when the file lacks the execute bit or has a broken shebang. A shebang with a trailing carriage return, which is what happens when a file is edited on Windows, produces exactly this and is invisible in an editor:

head -1 /srv/app/bin/start.sh | od -c | head -2   # look for \r \n

2. Relative paths and the working directory

Your app opens ./config/production.json and it works when you run it by hand from the project directory. Systemd starts with WorkingDirectory=/, so that path resolves to /config/production.json and does not exist.

WorkingDirectory=/srv/app

The same applies to relative paths in .env loading, SQLite files, and log destinations. This is the same class of wrong assumption as a build that works locally and fails in CI: the environment differs in a way nobody wrote down.

3. Environment variables that are simply not there

Your shell has them from .bashrc or a login profile. Systemd has almost nothing.

systemctl show myapp -p Environment
systemd-run --uid=appuser --pty /usr/bin/env    # what the service actually sees

Load a file explicitly:

EnvironmentFile=/etc/myapp/env

Note that EnvironmentFile is not a shell script. export FOO=bar is wrong there, quotes are treated literally in ways that surprise people, and command substitution does not happen.

4. Hardening directives that block the thing you need

226/NAMESPACE and mysterious permission errors usually come from the security directives people copy from a hardening guide:

ProtectSystem=strict      # entire filesystem read only
ProtectHome=true          # /home, /root, /run/user invisible
PrivateTmp=true           # your own /tmp, so nothing you wrote there exists
ReadWritePaths=/var/lib/myapp

PrivateTmp=true is the sneaky one. Anything your service writes to /tmp is invisible to everyone else and vanishes on restart, which produces bug reports that make no sense until you know.

Add back exactly what you need with ReadWritePaths rather than removing the protections.

5. It starts, then dies, then starts again

systemctl show myapp -p NRestarts
journalctl -u myapp --since "10 min ago" | grep -c "Started"

A service in a restart loop with Restart=always looks superficially alive. If Type= is wrong, systemd may also believe it started when it did not.

Type=simple means the process you exec is the service. Type=forking means it will daemonise and systemd should track the child. Declaring forking for a process that stays in the foreground makes systemd wait for a fork that never comes, then time out. Declaring simple for a process that daemonises makes systemd think it exited immediately.

Modern services should use Type=simple, or Type=notify if they can signal readiness with sd_notify. Do not daemonise under systemd.

The trick that saves the most time

Run the exact command, as the exact user, with the exact environment, without editing anything:

systemd-run --uid=appuser --gid=appgroup \
  --working-directory=/srv/app \
  --property=EnvironmentFile=/etc/myapp/env \
  --pty /usr/bin/node /srv/app/server.js

This reproduces systemd's conditions in a terminal where you can see the output immediately. Most 203s and permission problems become obvious within seconds.

Prevention

• Always absolute paths in ExecStart, always an explicit WorkingDirectory.
• systemd-analyze verify /etc/systemd/system/myapp.service catches syntax and dependency errors before you deploy.
• Remember systemctl daemon-reload after any unit edit. Editing the file and restarting is a no op for the changed settings, and it confuses people for a surprisingly long time.
• Add hardening directives one at a time, restarting between each, instead of pasting a block of twelve.
• Log to the journal rather than a file. StandardOutput=journal gives you journalctl -u correlation with the exit codes above, which is worth more than a separate logfile.