The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Feature #12598

Puppet should work correctly when the catalog has the deflate C-T-E applied.

Added by Garrett Honeycutt almost 3 years ago. Updated almost 2 years ago.

Status:AcceptedStart date:02/13/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:network
Target version:-
Affected Puppet version: Branch:
Keywords:compression catalog

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket may be automatically exported to the PUP project on JIRA using the button below:


Description

We have a customer that runs Puppet in the traditional Master and Agents configuration over a satellite network where they have extremely limited aggregate bandwidth and even less on a per node connection over the unicast link. This feature would allow one to optionally turn on catalog compression.


Related issues

Related to Puppet - Feature #12599: Puppet should apply the `deflate` C-T-E to data sent via ... Accepted 02/13/2012

History

#1 Updated by Garrett Honeycutt almost 3 years ago

similar to #12599 and #12560

#2 Updated by Chris Price almost 3 years ago

  • Category set to network
  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Nick Lewis

Nick Lewis looked into this and thinks it might not be that difficult… Nick, can you please add whatever insight you came up with?

#3 Updated by Nick Lewis almost 3 years ago

It appears that the agent will handle gzipped content for any request it makes using the REST terminus. This means it should be simple to put a gzip proxy in front of the master and have it just work.

Personally, I would rather not make this an option, preferring instead to just ensure we can handle gzipped requests and responses properly. That lets users set up their own compression if they want it, without need for us to provide it. Although, given there’s really no drawback to compressing everything, we could just turn gzip on for everything (or dynamically, based on the size/kind of content, who knows?). Perhaps turn it off in debug mode. But either way, I don’t see a good reason this should be a user-configurable option.

#4 Updated by Garrett Honeycutt almost 3 years ago

Compression has pro’s and con’s. The Pro is less bandwidth is being used and the Con is that takes CPU. Since catalog compilation is already CPU intensive, this might not be a good fit for everyone. My anecdotal field experience is that most people are CPU bound but have plenty of network bandwidth.

I am not positive if we should or should not ship with compression turned on, though we do need the ability the choose. Optimally you could also set the level of compression used.

#5 Updated by Anonymous almost 3 years ago

  • Subject changed from As someone with very limited network bandwidth, I would like to optionally enable compression of the catalog, so that we can scale our puppet environment to Puppet should work correctly when the catalog has the deflate C-T-E applied.
  • Status changed from Needs More Information to Accepted
  • Assignee deleted (Nick Lewis)

Nick Lewis wrote:

It appears that the agent will handle gzipped content for any request it makes using the REST terminus. This means it should be simple to put a gzip proxy in front of the master and have it just work.

It would be vastly easier to just have the web server – Apache or Nginx – that runs the master at any large scale deployment perform compression on the stream. No need to go to the trouble of adding an entire separate application to achieve the result.

Personally, I would rather not make this an option, preferring instead to just ensure we can handle gzipped requests and responses properly. That lets users set up their own compression if they want it, without need for us to provide it. Although, given there’s really no drawback to compressing everything, we could just turn gzip on for everything (or dynamically, based on the size/kind of content, who knows?). Perhaps turn it off in debug mode. But either way, I don’t see a good reason this should be a user-configurable option.

Absolutely. If we do this it should not be optional. I am happy to say that we should support this as retitled.

#6 Updated by Garrett Honeycutt almost 3 years ago

Glad to see that this is making progress! I’m not sure why this should not be optional, given the tradeoff of CPU for less bandwidth usage might negatively impact folks that are already constrained by CPU and have plenty of network IO. Also, wouldn’t we want to allow the user to set the compression level?

#7 Updated by Garrett Honeycutt almost 3 years ago

pinging for update and target version

#8 Updated by konrad rzentarzewski almost 2 years ago

i think it’s fixed but it wasn’t advertised. after setting “http_compression = true” on agent and “SetOutputFilter DEFLATE” in passenger on master i get a conversation that advertises “Accept-Encoding: gzip” on client and returns “Content-Encoding: gzip” (and gzipped catalog) from master.

tcpdump confirms that:

POST /production/catalog/lb2.arme.corp HTTP/1.0
Host: puppet
X-Real-IP: 192.0.2.102
X-Forwarded-For: 192.0.2.102
X-Client-Verify: SUCCESS
X-Client-DN: /CN=lb2.arme.corp
X-SSL-Subject: /CN=lb2.arme.corp
X-SSL-Issuer: /CN=stonka.arme.corp
Connection: close
Accept: pson, yaml, b64_zlib_yaml, dot, raw
Accept-Encoding: gzip; q=1.0, deflate; q=1.0; identity
Content-Type: application/x-www-form-urlencoded
Content-Length: 8165
facts_format=b64_zlib_yaml&facts=...

HTTP/1.1 200 OK
Date: Wed, 30 Jan 2013 12:46:31 GMT
Server: Apache/2.2.3 (CentOS) Phusion_Passenger/3.0.17
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.17
Status: 200
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: text/pson
^_<8b>^H^@^@^@^@^@^@...

Also available in: Atom PDF