The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Feature #4248

Load "library" plugins that are used by multiple puppet functions and delivered via pluginsync

Added by R.I. Pienaar almost 6 years ago. Updated about 3 years ago.

Status:AcceptedStart date:07/15/2010
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:plumbing
Target version:-
Affected Puppet version:0.25.5 Branch:
Keywords:autoloader

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Because we do not add all module lib directories to Ruby’s search path, ruby classes that exist only in the modules (because they have not yet been synced via pluginsync) cannot be found. The autoloader correctly loads files from the modules, but ruby itself can’t load supporting classes.

Old description:

At present we can copy parser functions, types and providers out with plugins sync, this works fine.

It seems though we cannot copy out Puppet::Util::* classes – useful for code accessed by various parser functions for example.

I need to do something like:

require File.dirname(__FILE__) + '/../../util/foo.rb'

from a parser function since:

require 'puppet/util/foo'

doesnt work.

I am also keen to be able to copy out new application classes that will tie into the puppet single executable thing, i guess these 2 features might be in the same realm.


Related issues

Related to Puppet - Feature #6522: Enable additional destinations & termini via plugins Accepted 03/01/2011
Related to Standard Library - Bug #13091: stdlib throws a LoadError when running with puppet apply Closed 03/13/2012
Related to Puppet - Bug #14073: puppet apply cannot find types in modules on Windows Duplicate 04/18/2012
Related to Puppet - Bug #15074: parent providers do not work with pluginsync Duplicate 06/15/2012
Related to Puppet - Feature #12126: make autoloader able to reload changed plugins Closed 01/24/2012
Related to Puppet - Feature #14149: The Puppet::Modules (or some other namespace) should be r... Closed 04/23/2012
Related to Puppet - Bug #7316: puppet face applications (subcommands) delivered via modu... Closed 05/02/2011
Related to Puppet - Bug #9827: Hooks for dependent settings not called Accepted 09/29/2011 10/05/2011
Related to Puppet - Bug #16651: Installing the cloud provisioner module breaks the node s... Duplicate 10/01/2012
Related to Puppet - Bug #18042: Deprecate face versions Accepted
Related to Puppet Documentation - Bug #16646: Don't instruct people to use #loadall within custom funct... Rejected 10/01/2012

History

#1 Updated by James Turnbull almost 6 years ago

  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Luke Kanies
  • Target version set to 2.6.1

+1 from me. Luke?

#2 Updated by Luke Kanies almost 6 years ago

  • Status changed from Needs Decision to Accepted
  • Assignee deleted (Luke Kanies)

I think this is related to another ticket, but I can’t find the ticket – basically, not all of the module ‘lib’ directories are added to Ruby’s search path. I thought Dan opened this ticket, but I couldn’t find it.

#3 Updated by R.I. Pienaar almost 6 years ago

And kind of related:

  • Give plugin code a way to have their options added to puppet.conf, maybe with a more freeform [section] in the conf

#4 Updated by James Turnbull over 5 years ago

  • Target version changed from 2.6.1 to 2.6.2

#5 Updated by James Turnbull over 5 years ago

  • Target version changed from 2.6.2 to 2.7.x

#6 Updated by Nigel Kersten over 5 years ago

This seems to clash with my bug report #4916

#7 Updated by R.I. Pienaar over 5 years ago

Really? I dont think I asked to replace existing Util modules, I want to add more. Ditto for applications, I dont want to replace puppet kick with another one, I want to add new applications.

#8 Updated by Nigel Kersten over 5 years ago

Yes. We hashed this out on IRC.

I misread this as replacing existing modules. My mistake.

Violent agreement all around.

#9 Updated by James Turnbull about 5 years ago

  • Category set to plumbing

#10 Updated by James Turnbull about 5 years ago

This seems really, really simple to me:

diff --git a/lib/puppet/util.rb b/lib/puppet/util.rb
index 850d147..cc0593d 100644
--- a/lib/puppet/util.rb
+++ b/lib/puppet/util.rb
@@ -28,16 +28,16 @@ module Util
     end
   end
 
-  
+
   def self.synchronize_on(x,type)
     sync_object,users = 0,1
     begin
-      @@sync_objects.synchronize { 
+      @@sync_objects.synchronize {
         (@@sync_objects[x] ||= [Sync.new,0])[users] += 1
       }
       @@sync_objects[x][sync_object].synchronize(type) { yield }
     ensure
-      @@sync_objects.synchronize { 
+      @@sync_objects.synchronize {
         @@sync_objects.delete(x) unless (@@sync_objects[x][users] -= 1) > 0
       }
     end
@@ -462,3 +462,4 @@ require 'puppet/util/execution'
 require 'puppet/util/logging'
 require 'puppet/util/package'
 require 'puppet/util/warnings'
+require 'puppet/util/plugins'
diff --git a/lib/puppet/util/plugins.rb b/lib/puppet/util/plugins.rb
new file mode 100644
index 0000000..ace2662
--- /dev/null
+++ b/lib/puppet/util/plugins.rb
@@ -0,0 +1,5 @@
+require 'rubygems'
+Gem.find_files('puppet/util/plugins/*.rb').each do |f|
+  require f
+end
diff --git a/lib/puppet/util/plugins/json.rb b/lib/puppet/util/plugins/json.rb
new file mode 100644
index 0000000..18a99c2
--- /dev/null
+++ b/lib/puppet/util/plugins/json.rb
@@ -0,0 +1,13 @@
+# Adds json formatted ouput for Puppet applications.
+# Usage: puppet agent --logdest json
+# Currently there's no way to load this as a plugin, see #4248. Appending this
+# method to puppet/util/log/destinations.rb works. Meh.
+# Would also be nice if Puppet::Util::Log had an #attributes method for 
+# accessors instead of instance_variable hacks.
+Puppet::Util::Log.newdesttype :json do
+  def handle(msg)
+    message = {}
+    msg.instance_variables.each {|v| message[v.sub("@","")] = msg.instance_variable_get(v) }
+    puts message.to_pson
+  end
+end

That’s obviously crude and lacks test and finesse. But it works.

#11 Updated by Luke Kanies about 5 years ago

I’m not sure if the specific failure. Can someone describe exactly what the problem is?

Do util files not get downloaded?

Are you expecting them to work from the module directly and they don’t?

Is there an exception somewhere?

#12 Updated by R.I. Pienaar about 5 years ago

Luke Kanies wrote:

I’m not sure if the specific failure. Can someone describe exactly what the problem is?

Do util files not get downloaded?

they get copied to the node alright, the master does not seem to have access to them though

If I put a parser function in modules/puppet/lib/puppet/parser/functions/test_func.rb the master will find it in its path, autoload it or whatever and i can just use it:

given the parser fun:

module Puppet::Parser::Functions
    newfunction(:test_func) do |args|
        require 'puppet/util/meh'
    end
end

and modules/puppet/lib/puppet/util/meh.rb

this fails in a node:

node default {
   test_func()
}

with:

err: no such file to load -- puppet/util/meh

Which suggests it finds the parser function in the module lib dirs but require fails

So I guess autloader needs to intercept require and it doesnt, or perhaps some documented helper method should be made that uses the autoloader to find this util class.

The other thing with applications, I just want to pluginsync out modules/puppet/lib/puppet/application/foo.rb and have it show up in the list of ‘puppet’

# ls -l /var/lib/puppet/lib/puppet/application/parse.rb
-rw-r--r-- 1 root root 2642 Feb 20 22:07 /var/lib/puppet/lib/puppet/application/parse.rb
Usage: puppet command 
Available commands are: agent, apply, cert, describe, doc, filebucket, kick, master, queue, resource

If instead I copy the same parse.rb into /usr/lib/ruby/site_ruby/1.8/puppet/application/parse.rb it just works:

# cp /var/lib/puppet/lib/puppet/application/parse.rb /usr/lib/ruby/site_ruby/1.8/puppet/application/parse.rb
# puppet
Usage: puppet command 
Available commands are: agent, apply, cert, describe, doc, filebucket, kick, master, parse, queue, resource

Are you expecting them to work from the module directly and they don’t?

Is there an exception somewhere?

#13 Updated by Luke Kanies about 5 years ago

R.I. Pienaar wrote:

[…] So I guess autloader needs to intercept require and it doesnt, or perhaps some documented helper method should be made that uses the autoloader to find this util class.

I think the problem is that ruby is not looking in the lib directory of the modules, so the right fix is probably just to add the libdir for every module into ruby’s loadpath.

The other thing with applications, I just want to pluginsync out modules/puppet/lib/puppet/application/foo.rb and have it show up in the list of ‘puppet’

This will work in 2.6.5 (and already works in the RCs). It’s required for my interfaces work, too.

#14 Updated by Luke Kanies about 5 years ago

  • Subject changed from Allow more flexibility in plugins to Plugin library dirs are not searched by ruby

#15 Updated by R.I. Pienaar about 5 years ago

Luke Kanies wrote:

R.I. Pienaar wrote:

[…] So I guess autloader needs to intercept require and it doesnt, or perhaps some documented helper method should be made that uses the autoloader to find this util class.

I think the problem is that ruby is not looking in the lib directory of the modules, so the right fix is probably just to add the libdir for every module into ruby’s loadpath.

here’s $: from a inline_template() of the master at the time:

/usr/lib/ruby/gems/1.8/gems/bson-1.0.4/lib/../lib
/usr/lib/ruby/gems/1.8/gems/mongo-1.0.5/lib/../lib
/usr/lib/ruby/gems/1.8/gems/gem_plugin-0.2.3/bin
/usr/lib/ruby/gems/1.8/gems/gem_plugin-0.2.3/lib
/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/bin
/usr/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib
/usr/lib/ruby/gems/1.8/gems/fastthread-1.0.1/bin
/usr/lib/ruby/gems/1.8/gems/fastthread-1.0.1/lib
/usr/lib/ruby/gems/1.8/gems/fastthread-1.0.1/ext
/usr/lib/ruby/gems/1.8/gems/cgi_mulby/1.8
/usr/lib64/ruby/site_ruby/1.8
/usr/lib64/ruby/site_ruby/1.8/x86_64-linux
/usr/lib/ruby/site_ruby
/usr/lib64/ruby/site_ruby
/usr/lib64/site_ruby/1.8
/usr/lib64/site_ruby/1.8/x86_64-linux
/usr/lib64/site_ruby
/usr/lib/ruby/1.8
/usr/lib64/ruby/1.8
/usr/lib64/ruby/1.8/x86_64-linux
.
/var/lib/puppet/lib

The other thing with applications, I just want to pluginsync out modules/puppet/lib/puppet/application/foo.rb and have it show up in the list of ‘puppet’

This will work in 2.6.5 (and already works in the RCs). It’s required for my interfaces work, too.

#16 Updated by Markus Roberts about 5 years ago

I think the problem is that ruby is not looking in the lib directory of the modules, so the right fix is probably just to add the libdir for every module into ruby’s loadpath.

Hmmm. Ruby’s load path is global and not thread safe. Multiple environments. Hmmm….

#17 Updated by Nigel Kersten about 5 years ago

I’m a bit concerned about environment leakage here too.

Will we be able to reliably ship two different versions of the same function/type/provider/whatever in different environments and be confident the correct version is used within environments?

#18 Updated by Luke Kanies about 5 years ago

Things that currently work across environments still will. Things that don’t (ruby classes, because constants are global) still won’t.

#19 Updated by donavan m about 5 years ago

I think there are two related issues in this ticket.

Loading a “library” plugin that’s used by multiple puppet functions. If it’s sync’d down to /var/lib/puppet/lib/puppet/util/foo.rb then a function with ‘require “puppet/util/foo”’ doesn’t load it as expected. I think most of the comments , $RUBYLIB etc, are addressing this.

Adding additional destinations for things like Puppet::Util::Log. These aren’t require’d by any functions, so need to get autoloaded by the application framework somehow. I only see Puppet::Util::Autoload.new instances for plugins in puppet/type, puppet/provider, puppet/parser/functions, & puppet/feature (?). James mentioned my particular use case in comment # 10 by just evaluating *.rb under the :plugindest directory.

I could have sworn there was also an open ticket something like “plugin paths shouldn’t be added to ruby $RUBYLIB”.

#20 Updated by Luke Kanies about 5 years ago

donavan m wrote:

I think there are two related issues in this ticket.

Loading a “library” plugin that’s used by multiple puppet functions. If it’s sync’d down to /var/lib/puppet/lib/puppet/util/foo.rb then a function with ‘require “puppet/util/foo”’ doesn’t load it as expected. I think most of the comments , $RUBYLIB etc, are addressing this.

I agree.

Adding additional destinations for things like Puppet::Util::Log. These aren’t require’d by any functions, so need to get autoloaded by the application framework somehow. I only see Puppet::Util::Autoload.new instances for plugins in puppet/type, puppet/provider, puppet/parser/functions, & puppet/feature (?). James mentioned my particular use case in comment # 10 by just evaluating *.rb under the :plugindest directory.

I’m not sure this has ever been asked for; it would be straightforward to add this as a point of extension. Is there a ticket file to this?

#21 Updated by Nigel Kersten over 4 years ago

This may be the same problem as #7316; we should investigate that before investing in a new solution here.

#22 Updated by Nigel Kersten over 4 years ago

  • Subject changed from Plugin library dirs are not searched by ruby to Load "library" plugins that are used by multiple puppet functions and delivered via pluginsync

#23 Updated by Chris Price almost 4 years ago

I believe that, as Nigel suggested, that this has been dealt with as part of the autoloader and pluginsync work that Patrick and I did for Telly (#7316, #11858, #12126, etc.).

We do not alter Ruby’s load path to reflect the quirks of our modulepath, but pluginsync will sync module content directly into the agent’s libdir (which is in Ruby’s load path) in a way that should allow the ruby ‘require’ statements to work as expected.

I would love for someone who is a stakeholder for this (R.I., Jeff) to try it out in Telly and let us know if my assessment is incorrect; otherwise I think that we can mark this as fixed in 3.0 and close this ticket.

#24 Updated by Anonymous almost 4 years ago

OK, so Chris and I are going to dive into this together on Friday.

As I mentioned in the comments many moons ago… This is not limited to pluginsync.

  • Puppet apply needs to work as a first class citizen. If it pluginsyncs, great, if not, it still needs to work.
  • Puppet master needs to work as a first class citizen. (Parser functions. Types and providers, etc…)

So, don’t go based on the subject alone. It’s not capturing the full scope of this problem.

#25 Updated by Nigel Kersten almost 4 years ago

apply does pluginsync now in 3.0, precisely to avoid the special casing of solutions for problems like this.

#26 Updated by eric sorenson over 3 years ago

Recording for posterity, later in the 3.0.0 release we stopped pluginsync on apply, because it clobbers the cache of downloaded plugins from the master with the (almost certainly more limited) set know to the modulepath of ‘puppet apply’. Gem directories (via #7788) and any lib/ subdirectories in modulepath get appended to $: to build up a consistent picture of loadable code without causing agent<->master disparities.

#27 Updated by Anonymous over 3 years ago

A correction on what Eric said:

lib/ subdirectories of modules DO NOT get added to $: in the Puppet 3.0.0 release. There was some experimentation with this that hacked the Kernel.require method much like Rubygems does, but that was not put in because of the issue of environments and a threaded environment. At the moment our only threaded environment (that I know of) is in webrick, if we can change webrick to not using threads, then that greatly mitigates some of the problems involved in this.

#28 Updated by Anonymous over 3 years ago

Bug #9827 shows the exact problem that is caused by not being able to load the library code from modules. The mount_providers module has a file lib/puppet/provider/mountpoint.rb that is loaded with require from lib/puppet/provider/mountpoint/linux.rb. This fails because the mountpoint.rb file is nowhere to be found in the load path.

#29 Updated by Josh Cooper over 3 years ago

  • Keywords set to autoloader

Andrew Parker wrote:

The mount_providers module has a file lib/puppet/provider/mountpoint.rb that is loaded with require from lib/puppet/provider/mountpoint/linux.rb. This fails because the mountpoint.rb file is nowhere to be found in the load path.

If the autoloader loads module A, and A requires module B, and both modules change, e.g. pluginsync, then the autoloader will never reload B, which will lead to crazy code inconsistency bugs. Fortunately, the agent forks (it will in 3.0.2 #17361) eliminating this issue on the agent side. But it is an issue on the master side, e.g. an autoloaded function requires utility code.

#30 Updated by Josh Cooper over 3 years ago

The proposed fix for #7316 will ensure an application can require custom facts, types, providers, applications, faces and actions from the modulepath or gem, and that code can in turn require utility code, also from the modulepath or gem. However, the proposed fix won’t fix this particular issue, for the master to require utility code from a function. Also true for report processors. Additional changes will be required to the compiler to ensure code does not leak across environments, likely fork (but not exec) a child to do the compile.

#31 Updated by Anonymous over 3 years ago

  • Target version deleted (2.7.x)

Also available in: Atom PDF