Apache, fastcgi, proxy_fcgi, and empty POST bodies with chunked transfer

Jeff Geerling | Aug. 24, 2017, 11:45 p.m. | Original

I've been working on building a reproducible configuration for Drupal Photo Gallery , a project born out of this year's Acquia Build Hackathon .

We originally built the site on an Acquia Cloud CD environment , and this environment uses a pretty traditional LAMP stack. We didn't encounter any difficulty using AWS Lambda to post image data back to the Drupal site via Drupal's RESTful Web Services API .

The POST request is built in Node.js using:

var reqPost = http.request(options_post, function(res) {
    res.on('data', function (chunk) {
        body += chunk;
    });
});
reqPost.write(JSON.stringify(jsonObject));
reqPost.end();

The way this kind of HTTP request works is using Transfer-Encoding: chunked . Chunked encoding is used in many cases when you need to send potentially large request bodies, and don't necessarily know the full size beforehand (e.g. you can't provide a Content-Length in the request).

Apache handles non-chunked transfers without issue, but in many cases, it seems Apache can't correctly pass on the request body if using Transfer-Encoding: chunked . Specifically, using PHP-FPM as the backend, I could not get the request body using either:

I was using the test PHP script and cURL command listed in this comment to test—basically, make a POST or PUT request with chunked encoding, and see what PHP gets via file_get_contents('php://input') .

To verify the server itself (pre-Apache) was receiving the request body, I used sudo ngrep "POST" tcp and port 80 to monitor incoming POST requests... and it indicated the data was definitely reaching the server (ruling out any AWS, DigitalOcean, or on-server firewall or proxy eating the data):

$ sudo ngrep "POST" tcp and port 80
interface: eth0 (165.227.96.0/255.255.240.0)
filter: (ip or ip6) and ( tcp and port 80 )
match: POST
####
T 54.237.220.121:40842 -> 165.227.102.121:80 [AP]
  POST /test.php HTTP/1.1..Content-Type: application/json..Host: gallery.jeffgeerling.com..Connection:
  close..Transfer-Encoding: chunked....2d..{"Bucket":"test-bucket","Name":"Test Object"}..
#######

So then I decided maybe something in Apache was eating the data. I had been using proxy_fcgi to route requests to PHP using CGI, as it seems to be the most modern and efficient approach when running Apache 2.4 and later, but I realized maybe that mod was breaking things.

So I switched to the older (but often-used) Apache fastcgi mod and configured it to route to PHP... and instead of having the POST body completely disappear, it gave a warning (so at least that was more helpful...):

<title>411 Length Required</title>
</head><body>
<h1>Length Required</h1>
<p>A request of the requested method POST requires a valid Content-length.<br />
</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at gallery.jeffgeerling.com Port 80</address>

So finally, I decided to switch to the old but reliable mod_php , which runs PHP inside Apache's process forks. And what do you know? The chunked request body arrived intact, and my problem was solved!

I'm guessing I could've also just switched to Nginx, which doesn't seem to suffer from the same CGI bug when proxying requests to PHP-FPM, but in this particular case, I wanted to make sure I was coming closer to replicating the earliest test environment we were using on Acquia Cloud.

During the course of my research on this issue, I ran into a few bug reports that seem slightly circular in nature:

The PHP bug comments are correct—it's not an issue with PHP, since PHP-FPM seems to handle things correctly with other web frontends (Nginx, maybe others?). But it's not encouraging seeing the Apache issue has been open since 2012, and even has a patch, but has zero reviews and only one comment in the past 5 years ?.

So the takeaway? If you need to handle chunked content transfer, don't rely on Apache with mod_proxy_fcgi or mod_fastcgi !

For more background and debugging on the issue, check out the original GitHub issue after which this blog post was written: Requests from Lambda logging "POST body is: null" .

Self-signed certificates via Ansible for local testing with Nginx Reverse-proxying a SOAP API accessed via PHP's SoapClient