Print  

This report will explain how we locked our Apache web server down to using only TLS encryption while blocking SSL, and how we finally got Apache on CentOS Linux to use all three versions of TLS (1.0, 1.1, 1.2). This work followed the alerts about the severe security weakness that was discovered in SSL version 3, which was named the Poodle vulnerability. We took immediate action to protect our customers' banking information by disabling all of our SSL encryption and switching entirely to the newer TLS encryption standard, because we take that whole banking security thing very seriously, for both our protection and yours. The more technical readers among you will be interested in knowing that we furthermore disabled the fallback mechanisms within our Apache web server, so that our legitimate users cannot have their software tricked into using breakable encryption, which is extremely important too. We did the right thing by responding to the problem swiftly, before there was widespread knowledge of it, so there is no indication that any of our customers had their credit card information stolen via their communications with our system. Of course we cannot speak for other Internet sites. Some of which had the bad encryption in place for quite a while afterward, and there are still some vulnerable sites which might expose confidential information to nefarious parties. So, before shopping at any site online, it is wise to test the security, and we proudly encourage you to check out our stellar rating. Yes, we've earned some bragging rights. It is also a very good idea not to shop with any company that is using Windows to power its servers. Our software is totally Linux-based, for speed, power, flexibility, security, and reliability. Everything that Microsoft software isn't, in other words.

The Mysterious PayPal IPN Problem After Killing SSLv3

We did what any responsible techs would have done. We killed SSL version 3 as soon as we became aware of its major security problem, and we expected to live happily ever after. It never really works that way, does it? The following is from our httpd.conf file, and it is how the problems started.

#       Added the following line to disable SSL version 3,
#       due to a serious vulnerability. 10/19/2014
#       Other SSL versions were already disabled by default.
SSLProtocol All -SSLv3

Everything seemed fine until we noticed that we were no longer being notified of any sales that had used PayPal as a checkout mechanism. PayPal's IPN system was no longer providing notifications to our system about successful checkouts. This was a bigger issue than may be immediately apparent, because customers tend to get angry when you just silently keep their money. Nobody wants that ugly call from Visa or its legal department either.

Anyway, everything else seemed to be working perfectly fine, so we had to initially conclude that the people at PayPal had fubared something in a panic to secure their banking system from the new SSL weakness. We and other people who were using OpenCart software for stores would only get ignored by them or canned responses, and absolutely no progress was made for weeks. Either the PayPal employees were as clueless to the cause as we were, or they simply did not care. We tweaked the curl parameters inside our code and conversed with OpenCart's author, Daniel Kerr, to no avail. The OpenCart user community was more-or-less useless, because there were too few who were able or willing to participate in the troubleshooting, so it became a mystery for us and Mr. Kerr to solve. We thought that surely PayPal was not actually requiring that insecure SSL be used for its IPN notices, and how scary is that? Of course, PayPal's online testing system for IPN  transactions was so totally broken that it reported consistently having successful communications with our store, which never actually happened. Since the PayPal IPN system was absolutely refusing to communicate with our store anymore, there was nothing to be gleaned from our log files, because nothing gets logged to nowhere. As seasoned techs, our most annoying work involves troubleshooting with nothing but a blind trail-and-error approach, when not having enough information to even make an educated guess. Success and failure in such cases becomes a matter of luck, so we sometimes breakout the prayers and hope that the Big Guy is well-enough versed in Internet technologies.

Our workaround was resorting to logging into PayPal the old fashioned way to see which of OpenCart's "Missing Orders" transactions had been successfully processed by PayPal, and then we manually changed the status of those transactions from "Missing Orders" to "Complete". Fortunately, our customers never saw an issue, as everything looked completely normal to them. Nevertheless, we warn other users of OpenCart-based stores that they should ignore all other transactions which are in a missing status, because these extra transactions (not seen at PayPal) are from people who never completed their checkout process.

How We Checked for Missing Orders in Our Version of OpenCart

1. From the administrative "Dashboard" (front page) we clicked the "Sales" menu at the top.
2. Then we selected "Orders".
3. On the resultant page, we clicked on the drop down menu under "Status" and selected "Missing Orders".
4. Finally, we clicked "Filter".

Where Did TLS version 1.1 and 1.2 Go?

Sarah's detective skills saved us once again. She eventually noticed from monitoring our site's use of encryption with different browsers on different operating systems that every encrypted connection was made using TLS version 1.0 only. Why wasn't any browser using the newer versions (1.1 and 1.2)? She tweaked the cipher priority and other pertinent Apache settings for days, but our server still refused to communicate with either TLS 1.1 or 1.2. We tested our server using the nmap utility, and found that our server did not even know about 1.1 or 1.2. Here's what we saw from an nmap test:

Blind Troubleshooting and Using The Force

We custom built our server with all of the right parameters, so where did TLS 1.1 and TLS 1.2 go? We pondered if this could be a part of the PayPal problem, and we started to become concerned about the fate of our online store when the web browsers inevitably begin the process of phasing out the oldest version of TLS. Thus, we had a future problem that needed to be proactively fixed before it shut us down, regardless of PayPal's behavior and its total aversion to cooperation with others. We proceeded to find the cause of our TLS issue, knowing that it might fix the PayPal problem too. We had all of the updated OpenSSL libraries on our system for our version of Linux (CentOS 6.6), including the ever-important developmental packages. We rebuilt our Apache web server, which incorporates the libraries, and there was still nothing except TLS 1. As you may have guessed, there was nothing logged anywhere that provided any useful information.

Our guess was that the numskulls who put together the CentOS packages for OpenSSL had somehow broken versions 1.1 and 1.2 of TLS. It sure tested fine with the command "openssl ciphers -v", but the darn thing still was not working. Perhaps only the developmental libraries are broken, and this could be why the program continues to work fine from the command line. So we endeavored to go to the source, literally. We downloaded the source code for the newest stable version of OpenSSL and then we built it ourselves. Next, we rebuilt our web server using this custom version of OpenSSL, and then Bingo! Now the server is suddenly offering all of the different flavors of TLS for whatever browsers can handle, with preference always being given to the newest and most secure version (1.2). Everything with PayPal works fine now, as it originally did before we disabled SSL.

Therefore, we have determined that PayPal's IPN system is using TLS 1.1 and/or TLS 1.2, but it refuses to communicate with TLS 1. From their ridiculous passive-aggressive behavior, we have to wonder if they consider this to be some kind of super-confidential security secret to be protected at all cost, and could their security department possibly be that clueless? Maybe we should just nervously change the subject and never bring this up again.

Tips for Doing It Yourself or Building a Better Apache

Order is very important. You must build OpenSSL and have it properly installed prior to building Apache and PHP. Apache will need to have the custom OpenSSL in place in order to use it to build its own SSL (TLS) module from it. Apache must be built and installed before PHP is built, because PHP's Apache module is built using Apache's compiler (apxs), and the derived PHP module has to match the existing Apache installation perfectly.

Thus, the mandatory order of operations for most servers will be: OpenSSL -> Apache -> PHP

Building and Installing OpenSSL

Unpack the newest stable version of OpenSSL in a temporary directory. Move into that directory and make sure it is clean by running "make clean". Then run the configuration with these parameters:

./configure --prefix=/opt/openssl-1.0.1j --openssldir=/opt/openssl-1.0.1j

Change  the "/opt/openssl-1.0.1j" part to reflect whatever directory you will be using on your system. In this case, we used the /opt directory structure to keep the new OpenSSL isolated from the regular packages which are already on the system. Finish the process with the usual commands: "make" and then "make install".

Building and Installing Apache

Apache has something of a custom build process, so it is not just a matter of "./configure", "make", and "make install". Do these commands before running Apache's "make" process:

make clean
./buildconf

Next, you are going to want to make your own configuration script considering all of the options that you are likely going to need to build a desirable version of Apache. The script must contain these parameters:

--enable-ssl
--enable-ssl-staticlib-deps
--with-ssl=/opt/openssl-1.0.1j
--enable-static-ssl

Notice the line with "/opt" (--with-ssl=) matches the equivalent line from the build of OpenSSL. It must match exactly. You are going to have a real problem if Apache cannot find your new version of OpenSSL. It is likely to fallback to using the same broken libraries that were originally the problem, but it might not build at all. As a reference, here is the actual script used to build our awesome Apache web server:

When you have finished configuring Apache thusly, you may proceed to finish the process with "make" and "make install". For the newer Apaches, it should install itself in /usr/local/apache2 by default. You'll be finished in the unlikely case that you do not need PHP for anything, but plug ahead otherwise.

We additionally recommend these lines inside your httpd.conf file to push the preference of TLS 1.2 and ciphers that are 256 bit.

Building and Installing PHP

Unpack the PHP source into a temporary directory and go into it. Before configuring PHP, we recommend that you issue the following commands:

make clean
rm -f config.cache

Make sure to include this line when you run configure: "--with-apxs2=/usr/local/apache2/bin/apxs". For example:

./configure --with-apxs2=/usr/local/apache2/bin/apxs

Here is the PHP configuration script used for our main server (this one) as an example.

Troubleshooting

Broken MySQL Database Problem

MySQL passwords may stop working with a PHP upgrade. Let's not go into long explanations, because the situation involves the PHP designers being asses and deciding what is best for us. They want us to know that they are of a superior intellect, by the way, in a similar fashion to that infamous SystemD and Pulse Audio guy. You really don't want to know, because it will just make you angry and ruin your day. What you do actually want to know is how to fix their intentionally-induced problem, of course. The fix involves finding your my.cnf file (probably in /etc). Add this line to the main "[mysqld]" section of the file: "old_passwords = 0". Make sure there is no line like this that puts it equal to "1". Completely restart MySQL. Don't worry, for your downtime should only be like 1/2 second. Then resubmit or change your passwords for every user needing PHP access to MySQL. It will probably be every user except for root on most systems. The passwords do not necessarily need to be something new. They just need to be re-inserted into MySQL. This can be done using phpMyAdmin by going to "Privileges" and editing the user passwords. Be sure to select password hashing to be "MySQL 4.1+". We did nothing with our root password for MySQL, because this blocks PHP from that account completely.