The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #19884

Intermittent SSL errors in PuppetDB <-> master communication

Added by Hugh Cole-Baker about 3 years ago. Updated over 2 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:1.4.0
Keywords: Affected PuppetDB version:1.1.1
Branch:https://github.com/puppetlabs/puppetdb/pull/512

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

We are seeing occasional errors on the Puppet master where it fails to compile catalogs for nodes, or fails to submit ‘replace facts’ or ‘replace catalog’ commands for nodes, and reports an error in the logs like this:

Mar 24 22:51:51 puppet-1 puppet-master[22839]: Could not retrieve resources from the PuppetDB at puppet-1.our.domain:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A on node deploy.our.domain

This error seems to be accompanied by one in the PuppetDB logs like this:

2013-03-24 22:51:51,014 WARN  [qtp1193921293-37] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid TLS padding data

It only tends to happen intermittently, and if the agent retries the same operation to retrieve its catalog then it generally succeeds.

The master and PuppetDB processes are running on the same (Amazon EC2) server and PuppetDB is using a PostgreSQL 9.1 database on a separate server for data storage. The server is running Ubuntu 12.04.1, some other versions of packages installed that might be involved are:

Puppetmaster: 3.1.1-1puppetlabs1
PuppetDB:     1.1.1-1puppetlabs1
OpenSSL:      1.0.1-4ubuntu5.7
Ruby:         1.8.7.352-2ubuntu1.1
OpenJDK:      7u15-2.3.7-0ubuntu1~12.04.1

History

#1 Updated by Ken Barber about 3 years ago

puppet-users discussion here: https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/M2tYq-hpKeQ

So I found the patch that introduced this error message: http://cr.openjdk.java.net/~ewendeli/7u15-openjdk/jdk/src/share/classes/sun/security/ssl/CipherBox.java.udiff.html:

changeset:   5799:068448362d88
user:        wetmore
date:        Thu Feb 07 11:48:13 2013 -0800
summary:     8006777: Improve TLS handling of invalid messages

It seems to be a security fix, as the bug is hidden in the Oracle database. I bet it is related to this CVE: http://packetstormsecurity.com/files/cve/CVE-2013-0169.

The patch was introduced in 7u15 by the looks of it.

Someone had some success downgrading to OpenJDK 6. I’d imagine downgrading to 7u14 would also solve it, but as this is a security patch it might be insecure to downgrade. Downgrading to the latest OpenJDK 6 however, should include the CVE patch.

#2 Updated by Russ Parsloe about 3 years ago

Just to confirm what Hugh is saying, I too get the same errors. My versions of software are all but identical:

Ubuntu:        12.04.2 LTS
Puppet Master: 3.1.0-1puppetlabs1 
Puppet DB:     1.1.1-1puppetlabs1 
OpenSSL:       1.0.1-4ubuntu5.8 
Ruby:          1.8.7.352-2ubuntu1.2 
OpenJDK:       7u15-2.3.7-0ubuntu1~12.04.1

Is this a bug, or a case of downgrading Java to remove the warning from being displayed?

#3 Updated by Ken Barber about 3 years ago

Russ Parsloe wrote:

Just to confirm what Hugh is saying, I too get the same errors. My versions of software are all but identical:

[…]

Is this a bug, or a case of downgrading Java to remove the warning from being displayed?

We think its a bug in OpenJDK. Downgrading OpenJDK to 6 we believe should solve the problem for now.

#4 Updated by Ken Barber about 3 years ago

Still awaiting confirmation that openjdk-6 solves it for people.

Also found this alternate that I sent to the mailing list:

An alternative thing to try – I found this in the openssl changelog: http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.0.1-4ubuntu5.8/changelog. Looks like the patch for CVE-2013-0169 was reverted due to a bug, but it has now been re-enabled with a fix for the regression. Can you try upgrading to 1.0.1-4ubuntu5.8 (combined with openjdk-7) to see if this helps?

I’ve tried to replicate the issue on my local Ubuntu 12.04 box but so far no good, so I need someone who is getting the issue to confirm all this.

#5 Updated by Ken Barber about 3 years ago

  • Status changed from Unreviewed to Accepted

#6 Updated by Hugh Cole-Baker about 3 years ago

I’ve tried upgrading to openssl 1.0.1-4ubuntu5.8 (along with using openjdk-7) but the errors still occasionally appear. I’m going to try downgrading to openjdk-6 temporarily, so I’ll let you know how that goes.

#7 Updated by Ken Barber about 3 years ago

Sorry – I should have been clear, but did you try restarting Apache (or just puppetmaster if you aren’t using apache/passenger) after you upgraded openssl? Just want to be sure.

Hugh Cole-Baker wrote:

I’ve tried upgrading to openssl 1.0.1-4ubuntu5.8 (along with using openjdk-7) but the errors still occasionally appear. I’m going to try downgrading to openjdk-6 temporarily, so I’ll let you know how that goes.

#8 Updated by Ken Barber about 3 years ago

Hugh Cole-Baker wrote:

I’ve tried upgrading to openssl 1.0.1-4ubuntu5.8 (along with using openjdk-7) but the errors still occasionally appear. I’m going to try downgrading to openjdk-6 temporarily, so I’ll let you know how that goes.

Any luck with the downgrade Hugh?

#9 Updated by Chuck Schweizer about 3 years ago

I see this using oracle java also:

RHEL 6.4 java-1.7.0-oracle-1.7.0.17-1jpp.1.el6_4.x86_64

Downgrading back to 1.6 seems to make the error go away. Should puppetdb specify the path to the java binary instead of using /usr/bin/java. eg. /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java

#10 Updated by Hugh Cole-Baker about 3 years ago

Ken Barber wrote:

Hugh Cole-Baker wrote:

I’ve tried upgrading to openssl 1.0.1-4ubuntu5.8 (along with using openjdk-7) but the errors still occasionally appear. I’m going to try downgrading to openjdk-6 temporarily, so I’ll let you know how that goes.

Any luck with the downgrade Hugh?

I ran PuppetDB on OpenJDK 6 for a day and didn’t see any of these errors, so I’m going to stick with that for PuppetDB for the time being.

#11 Updated by Stefan Schulte about 3 years ago

I was also seeing the error on RHEL6. Switching from OpenJDK7 back to OpenJDK6 worked for me.

#12 Updated by Deepak Giridharagopal almost 3 years ago

I’m wondering if at this point we should modify our packages to specify a dependency on just jdk6…

#13 Updated by Deepak Giridharagopal almost 3 years ago

Okay, looks like the problem is with Diffie-Hellman ciphers on the JDK side. I’ve got a prototype patch that disables those ciphers:

https://gist.github.com/grimradical/9755ac35c64f519e1b49

I’m using the following horrible script to repeatedly open SSL connections to PuppetDB, which demonstrates (occasionally) a handshake error:

#! /bin/bash
count=0
while [ 1 -eq 1 ]
do
echo "Hello" | openssl s_client -key privkey.pem -cert pubkey.pem -CAfile ca_crt.pem -connect localhost:8081 >/dev/null
if [ $? -ne 0 ]; then
exit 1
fi
((count+=1))
echo $count
done

The CA/certs/keys are things that PuppetDB has been configured to accept. Anyways, with the patch applied, the problem goes away. Among the ciphers I’m allowing in the patch are several that are approved by NIST and conform to US FIPS security standards, FWIW.

Problem is, I think we should only disable DHE on jdk’s that exhibit the problem. 1.7u5 and earlier seem fine, but I don’t know which more recent ones are affected. Need more testing.

#14 Updated by Deepak Giridharagopal almost 3 years ago

  • Status changed from Accepted to In Topic Branch Pending Review
  • Branch set to https://github.com/puppetlabs/puppetdb/pull/512

#15 Updated by Ken Barber almost 3 years ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release
  • Target version set to 1.4.0

#16 Updated by Ken Barber over 2 years ago

  • Status changed from Merged - Pending Release to Closed

Also available in: Atom PDF