Back to Blog

500 Internal Server Error, readv() failed (104: Connection reset by peer) while reading upstream [Solved]

March 24, 2016

By Matt Mombrea
No Comments

Running Nginx and PHP-FPM is a pretty great setup for the most part. One thing we’ve struggled with on occasion however are some problems with unclear error messages and non-obvious solutions. One of those instances involved the following generic error message in the nginx logs:

500 Internal Server Error,readv() failed (104: Connection reset by peer) while reading upstream

That’s a pretty broad error message used to indicate all types of problems and usually you look elsewhere for a clue, like the php logs. Unfortunately, none of the other system logs had any information related to the error.

We started seeing this message immediately after a distribution upgrade from Debian 7 Wheezy to Debian 8 Jessie which made the scope of the problem very large. The strangest symptom was that PHP applications would work intermittently. One request would execute fine, the next page load would fail with the 500 error. Restarting nginx and php might allow the first few requests to work and then it would start failing again.

We tried a lot of things to fix this, pouring over the usual culprits like listener pool exhaustion, file permission and ownership, etc. before the real problem was discovered.

Opcode Cache

When the server was running Debian 7 we had XCache configured for the opcode cache on PHP 5.4 (the stable version for Debian 7). With the upgrade to Debian 8, PHP was upgraded to 5.6 and, along with it, the Zend opcode cacher was installed and enabled by default. The two opcode caches went to war behind the scenes, wreaking havoc on PHP applications.

4814101

In case you’re not familiar with what an opcode cache does, here’s the short version. Since PHP is an interpreted language, the system reads PHP code on demand and compiles it down to machine code for each request to run the application. An opcode cacher optimizes this process by caching the compiled code and storing it in RAM where it can be reused without recompiling on the next request. This greatly improves the performance of PHP applications by making it behave more like a compiled application.

Based on the above description you can imagine why having two opcode caches running simultaneously would cause problems. It’s a crap shoot as to which engine had cached the compiled code and which will serve up the result for a request. This has the effect of delivering a broken / partially compiled application and causing the 500 server error.

The Fix

To identify the issue, simply execute the following command at the terminal and read the output:

php -v

If you see more than one cache system listed, you’re going to have a bad time.

Just one opcode system listed here as it should be

Just one opcode system listed here as it should be

When the problem was happening, XCache was also listed here along side the Zend OPcache.

The solution is simple enough, uninstall one! I chose to remove my trusty XCache in favor of Zend since it now came bundled with the PHP package in Debian 8.

apt-get remove php5-xcache --purge

And whamo, everything was instantly back to normal.

Matt Mombrea

Matt Mombrea

Matt is a longtime entrepreneur and software engineer. He also sits on the board of Computer Science at SUNY Fredonia. Born and raised in Buffalo, Matt is passionate about building and maintaining great businesses in Buffalo. Apart from leading Cypress North, Matt architects our company’s datacenter, engineers dozens of custom applications, and directs the rest of the development team.

See Matt's Most Recent Posts

Share this post

Leave a Reply

Search our blog

Start A Project

Categories

What's next?

Well...you might like one of these

Article

Is Google Analytics 4 a Tactical Move...

There’s something fishy going on with the way that Google...

Read article

Article

How We Build It: Website Process with...

This month, I sat down with Matt Mombrea, Chief Technical...

Read article

Article

[SOLVED] Using an in-memory repository....

.NET Core Data Protection One of the main benefits of...

Read article