|
Technical Deep Dive

Debugging "Killed" Errors on Linux: When Node.js Won't Run

🕒 15 min read📅 December 2024🏷️ Node.js, Linux, DevOps

The Problem

A mysterious issue appeared on an Ubuntu 20.04 server running on Azure where every Node.js-related command was immediately terminated with a simple "Killed" message:

bash
root@www:~# pm2 status
Killed
root@www:~# npm install next@14
Killed
root@www:~# node --version
Killed

Even basic commands like watch were being killed, making it nearly impossible to diagnose the issue.

Initial Assumptions (All Wrong!)

When processes get killed on Linux, the typical culprits are:

Out of Memory (OOM) Killer

The kernel killing processes due to RAM exhaustion

Swap Space Issues

No swap configured to handle memory spikes

Resource Limits

cgroups or ulimits restricting processes

The Real Problem

A Corrupted Node.js Installation

After extensive debugging, we discovered the root cause wasn't system resources at all. The issue was:

Node.js v20.19.5 installed via NVM was corrupted or incompatible, causing the binary itself to crash immediately upon execution.

Key Evidence:

  • Simple bash commands worked fine (for loops, background processes)
  • Only Node.js, npm, pm2, and watch were killed
  • System Node v10 from apt worked (but was too old)
  • curl binary was "installed" but missing from filesystem

The Solution: Clean Reinstall

Step 1: Remove All Corrupted Node Installations

bash
# Remove apt-installed Node (if present and old)
apt-get remove -y nodejs nodejs-doc libnode64

# Remove all global npm/node installations
rm -rf /usr/local/lib/node_modules
rm -rf /usr/local/bin/npm
rm -rf /usr/local/bin/npx

# Remove NVM installations
rm -rf ~/.nvm

Step 2: Install Node.js from Official Binary

Instead of using package managers or NVM, we used the official pre-compiled binary:

bash
# Download Node 18 LTS
cd /tmp
curl -O https://nodejs.org/dist/v18.20.5/node-v18.20.5-linux-x64.tar.xz

# Extract
tar -xf node-v18.20.5-linux-x64.tar.xz

# Install to /usr/local
cp -r node-v18.20.5-linux-x64/* /usr/local/

# Update PATH
export PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Step 3: Verify Installation

bash
which node
# Output: /usr/local/bin/node

node --version
# Output: v18.20.5

npm --version
# Output: 10.8.2

✅ Everything now worked perfectly!

Key Takeaways

For Troubleshooting "Killed" Errors:

  • 1.Don't assume OOM - check dmesg and journalctl to confirm
  • 2.Test the binary directly with the simplest command possible
  • 3.Check for filesystem corruption using file and ls commands
  • 4.Sometimes a clean slate is faster than debugging

For Node.js Installations:

  • 1.Official binaries are most reliable for production servers
  • 2.Avoid multiple installation methods - pick one and stick with it
  • 3.Document your PATH to know which binary is being called
  • 4.Always verify with --version commands immediately after install

Conclusion

The "Killed" error is one of the most frustrating issues to debug because it provides almost no information. In this case, what appeared to be a memory issue was actually a corrupted Node.js binary. The solution wasn't adding resources or adjusting limits—it was completely removing the problematic installation and reinstalling from a clean, official source.

When facing similar issues, remember: sometimes the best debugging approach is elimination. Remove the variables, start fresh, and build up from a known-good state.

Have you encountered similar issues?

Share your experience and help others in the community!

Tags: Node.js • Linux • DevOps • Troubleshooting • Ubuntu • Azure

© 2024 Technical Blog • All rights reserved