500 Internal Server Error, readv() failed (104: Connection reset by peer) while reading upstream [Solved]

cn-500-internal-server-error

Running Nginx and PHP-FPM is a pretty great setup for the most part. One thing we've struggled with on occasion however are some problems with unclear error messages and non-obvious solutions. One of those instances involved the following generic error message in the nginx logs:

500 Internal Server Error,readv() failed (104: Connection reset by peer) while reading upstream

That's a pretty broad error message used to indicate all types of problems and usually you look elsewhere for a clue, like the php logs. Unfortunately, none of the other system logs had any information related to the error.

We started seeing this message immediately after a distribution upgrade from Debian 7 Wheezy to Debian 8 Jessie which made the scope of the problem very large. The strangest symptom was that PHP applications would work intermittently. One request would execute fine, the next page load would fail with the 500 error. Restarting nginx and php might allow the first few requests to work and then it would start failing again.

We tried a lot of things to fix this, pouring over the usual culprits like listener pool exhaustion, file permission and ownership, etc. before the real problem was discovered.

Opcode Cache

When the server was running Debian 7 we had XCache configured for the opcode cache on PHP 5.4 (the stable version for Debian 7). With the upgrade to Debian 8, PHP was upgraded to 5.6 and, along with it, the Zend opcode cacher was installed and enabled by default. The two opcode caches went to war behind the scenes, wreaking havoc on PHP applications.

4814101

In case you're not familiar with what an opcode cache does, here's the short version. Since PHP is an interpreted language, the system reads PHP code on demand and compiles it down to machine code for each request to run the application. An opcode cacher optimizes this process by caching the compiled code and storing it in RAM where it can be reused without recompiling on the next request. This greatly improves the performance of PHP applications by making it behave more like a compiled application.

Based on the above description you can imagine why having two opcode caches running simultaneously would cause problems. It's a crap shoot as to which engine had cached the compiled code and which will serve up the result for a request. This has the effect of delivering a broken / partially compiled application and causing the 500 server error.

The Fix

To identify the issue, simply execute the following command at the terminal and read the output:

php -v

If you see more than one cache system listed, you're going to have a bad time.

Just one opcode system listed here as it should be
Just one opcode system listed here as it should be

When the problem was happening, XCache was also listed here along side the Zend OPcache.

The solution is simple enough, uninstall one! I chose to remove my trusty XCache in favor of Zend since it now came bundled with the PHP package in Debian 8.

apt-get remove php5-xcache --purge

And whamo, everything was instantly back to normal.

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Meet the Author

mmombrea-headshot
CTO / Partner

Matthew Mombrea

Matt is our Chief Technology Officer and one of the founders of our agency. He started Cypress North in 2010 with Greg Finn, and now leads our Buffalo office. As the head of our development team, Matt oversees all of our technical strategy and software and systems design efforts.

With more than 19 years of software engineering experience, Matt has the knowledge and expertise to help our clients find solutions that will solve their problems and help them reach their goals. He is dedicated to doing things the right way and finding the right custom solution for each client, all while accounting for long-term maintainability and technical debt.

Matt is a Buffalo native and graduated from St. Bonaventure University, where he studied computer science.

When he’s not at work, Matt enjoys spending time with his kids and his dog. He also likes to golf, snowboard, and roast coffee.