My Overloaded Server StoryDecember 7th, 2015 • filed under Programming
So as you probably know by now, I converted my blog to Jekyll, yesterday, and it’s been a huge success, in my opinion. I’m motivated more than ever to start blogging more because I have the added tinker factor and using Git and GitHub to keep everything organized rocks.
But that isn’t what this story is about. Tonight’s story is about a server. A lowly VPS floating somewhere in the SFO DigitalOcean…. ocean. I’ve had this server running for many months–probably over a year now and I just don’t know it–and it’s hosted just about anything and everything I’ve tinkered with including many a failed idea.
I use this server a lot and for the most part it always responds well and isn’t sluggish. It’s hosted my former WordPress blog since January and hasn’t made but a peep about it so I’ve spent quite a bit of time thinking nothing’s amiss. I felt, though, that Jekyll shouldn’t be taking as long as it was to build my site but I couldn’t convince myself the server was overloaded.
As it Turns OutTM, I was wrong.
My baby of a server has, for the last five or so months, been filling Newrelic graphs with stuff like this:
So I’m just going to go ahead and say that’s not good. The only graph above that’s even remote decent is the Disk graph, but it’s washed over by the CPU > 80% indication so it’s also pretty much hosed.
At this point I’m honestly surprised. I’m a terrible pseudo-sysadmin and I should be fired but there’s no one to fire me and I’m the boss so whatever. If I can fix all this, I’m giving myself a raise.
I started digging. I wanted to see what’s running and who’s sucking up all the juices. I fired up
top and waited a few seconds. A few things popped up: a couple instances of
node and an instance of
Hmm. That doesn’t make any sense… I’m not running any node apps and jekyll is the only ruby thing I use right now… oh wait.
See, I tinkered with NodeJS apps late last spring. Ghost was a blogging tool I was considering for a while. I also tinkered with Discourse to see what it was all about. Turns OutTM, neither are suited for my needs.
I don’t really know what happened but I can only assume I forgot about them and they’ve been running all this time. I had a total of five Node instances running and Docker was running Discourse so between the two of them, I was on
swap 24/7. So dirty.
This story isn’t super climactic in any way and the fix was easy: kill all the things. I also found this to be a good time to remove Node and Docker since I need neither.
I still couldn’t figure out why my disk usage was so high, though. Leave it to me to forget about a 4GB
.iso I left in a folder and leave it to Discourse to fail at sending 5GB of emails through postfix over the last five months and clog up my
/var/mail folder. Needless to say, after the almost-winter cleaning, I gained quite a bit of ground:
Sorry this story wasn’t more interesting. If you’re curious, my Jekyll build time was cut in half.
That’s about it. Orphaned tools and processes makes my server a dull VPS.