The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #4409

puppetmasterd does not find custom types for environment

Added by Rudy Gevaert almost 6 years ago. Updated over 2 years ago.

Status:ClosedStart date:07/30/2010
Priority:UrgentDue date:
Assignee:-% Done:

0%

Category:plumbing
Target version:-
Affected Puppet version:2.6.0 Branch:http://github.com/MarkusQ/puppet/tree/ticket/2.6.x/4409
Keywords: customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Hello,

I can’t get plugins in modules to work. I tried IRC and puppet-users (http://groups.google.com/group/puppet-users/browse_frm/thread/1fa5616448b1ffad) , all led to no help/solution :(. This is blocking me to deploy puppet in our infrastructure.

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type vcsrepo at /etc/puppet/development/manifests/nodes.pp:37 on node puptest.ugent.be

I have attached several attachments that show my config.

thank you in advance

client.trace - trace of running the puppet agent (7.23 KB) Rudy Gevaert, 07/30/2010 10:07 am

tree.libdir - contents of the synced plugins (376 Bytes) Rudy Gevaert, 07/30/2010 10:07 am

node_definition - definition of the node (187 Bytes) Rudy Gevaert, 07/30/2010 10:07 am

tree.master - directory tree on the puppetmaster (14.3 KB) Rudy Gevaert, 07/30/2010 10:07 am

puppet.conf.master - puppetmaster config (939 Bytes) Rudy Gevaert, 07/30/2010 10:07 am

puppet.conf.client - node config (250 Bytes) Rudy Gevaert, 07/30/2010 10:07 am

output (12.9 KB) Rudy Gevaert, 09/03/2010 08:32 pm

issue4409.txt Magnifier (19.9 KB) John Warburton, 11/18/2010 04:20 am


Related issues

Related to Puppet - Bug #1175: Custom function doesn't work when using multiple environm... Closed
Related to Puppet - Bug #4260: A2mod can not be depended on in automated runs Closed 07/16/2010
Related to Puppet - Feature #6907: Ensure providers can be used in the same puppet run that ... Closed 03/30/2011
Related to Puppet - Bug #13858: Custom types in environments require loading into master'... Closed 04/08/2012
Related to Puppet - Feature #11900: Dynamic environment interpolation in puppet master config... Needs More Information 01/11/2012 05/01/2012
Related to Puppet - Bug #18154: Master and agent should not share libdir Accepted

History

#1 Updated by James Turnbull almost 6 years ago

  • Status changed from Unreviewed to Investigating
  • Assignee set to Dan Bode

Dan – any chance you can give this chap a hand?

#2 Updated by Dan Bode almost 6 years ago

can you verify if this problem also exists with 0.25.5?

#3 Updated by Rudy Gevaert over 5 years ago

Hi Dan and James, thanks for looking into this.

I have installed 0.25.5 (deb package from squeeze) on master and client and get this output:

/usr/lib/ruby/1.8/puppet/application.rb:217:in `run'
/usr/sbin/puppetd:160
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of resource: Could not retrieve information from source(s) puppet://puppet.ugent.be/plugins
notice: /File[/var/lib/puppet/lib/facter]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/facter]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/facter/configured_ntp_servers.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/facter/configured_ntp_servers.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/a2mod]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/a2mod]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/a2mod/a2mod.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/a2mod/a2mod.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/bzr.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/bzr.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/cvs.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/cvs.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/git.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/git.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/hg.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/hg.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/svn.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/svn.rb]: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/type]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/type]: Skipping because of failed dependencies
notice: /Filecommit:/var/lib/puppet/lib/puppet/type/a2mod.rb: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /Filecommit:/var/lib/puppet/lib/puppet/type/a2mod.rb: Skipping because of failed dependencies
notice: /File[/var/lib/puppet/lib/puppet/type/vcsrepo.rb]: Dependency file[/var/lib/puppet/lib] has 1 failures
warning: /File[/var/lib/puppet/lib/puppet/type/vcsrepo.rb]: Skipping because of failed dependencies
debug: Finishing transaction 69939312083200 with 0 changes
info: Loading facts in configured_ntp_servers
info: Loading facts in configured_ntp_servers
debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; using pson
info: Caching catalog for puptest.ugent.be
debug: Creating default schedules
debug: Loaded state in 0.01 seconds
info: Applying configuration version '1280753718'
debug: Finishing transaction 69939313750120 with 0 changes
debug: Storing state
debug: Stored state in 0.03 seconds
notice: Finished catalog run in 0.04 seconds

#4 Updated by Dan Bode over 5 years ago

check the servers modulepath: #puppetmasterd —configprint modulepath

also check all of the lib directories accessible in the modulepath and ensure that they are readable by puppet:puppet

#5 Updated by Rudy Gevaert over 5 years ago

wopr:~# puppetmasterd —configprint modulepath;puppetmasterd —environment development —configprint modulepath; puppetmasterd —environment production —configprint modulepath /etc/puppet/production/modules /etc/puppet/development/modules /etc/puppet/production/modules

I double checked the permissions and all are fine.

thanks in advance

#6 Updated by Dan Bode over 5 years ago

i think it might have something to do with the combination of plugins and environments which is a known issue on .2.5.x. try to catch me on irc (bodepd). in the mean time, I will see if I can recreate.

#7 Updated by Dan Bode over 5 years ago

  • Subject changed from plugins in modules Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type to puppetmasterd does not find custom types for environment
  • Priority changed from Normal to High

I have verified this. The problem is that the puppetmaster seems to only look for plugins (esp custom types) in its own modulepath and libdir.

recreate: – set up an environment on the puppetmasters puppet.conf

     
[env1]
  modulepath=/etc/puppet/env1/

then add the vcsrepo module to that environment’s modulepath (and not to the puppetmasters libdir or moduleapath)

/etc/puppet/env1/vcsrepo/…

now on the client, set

[agent]
  pluginsync=true 

and run

puppet agent -t --environment env1
info: Retrieving plugin
info: /File[/var/lib/puppet/lib]: Storing newly-audited value  for content
info: Loading facts in certname
info: Loading facts in certname
dnsdomainname: Unknown host
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type vcsrepo at /etc/puppet/manifests/site.pp:6 on node puppetclient
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

the puppetmaster cannot find the custom resource type b/c it does not appear to be looking in the modulepath of the agents environment.

I am escalating this to high since I thought plugins would work with environments on 2.6.x.

#8 Updated by Dan Bode over 5 years ago

  • Assignee changed from Dan Bode to Jesse Wolfe

#9 Updated by Nigel Kersten over 5 years ago

I was under the impression this was supposed to have been fixed for 2.6 ?

#10 Updated by Markus Roberts over 5 years ago

I was under the impression this was supposed to have been fixed for 2.6 ?

Thus the “investigating”; if you’re aware of any related tickets / places where it was believed to be fixed, or have any other insights feel free to add them here.

#11 Updated by Nigel Kersten over 5 years ago

Yep, that’s the plan. I’ll poke through the commits as I was sure I’d seen something fly by that was supposed to resolve this.

#12 Updated by Dan Bode over 5 years ago

  • Assignee deleted (Jesse Wolfe)

#13 Updated by Rudy Gevaert over 5 years ago

hi guys. Not that I want to be rude, but I was wondering if this is still on the roadmap for 2.6.1… Thanks!

#14 Updated by James Turnbull over 5 years ago

At this stage I’d suggest it’s unlikely it’ll make 2.6.1 but I will discuss with Engineering today.

#15 Updated by Nigel Kersten over 5 years ago

There are a couple of reasons I think this is a high priority issue that deserves attention.

  • Setting a default modulepath is undesirable if you want undefined environments to simply fail rather than fall back to a default.
  • Having a single location for plugin lookup on the server is exceedingly problematic if you want a release process for plugins where the acceptable parameters for a type may change.
  • Usability. This seems to bite a fair few people when they’re getting started with environments.

#16 Updated by James Turnbull over 5 years ago

  • Category set to plumbing
  • Target version set to 2.6.1

#17 Updated by Markus Roberts over 5 years ago

  • Assignee set to Markus Roberts

Rudy — no worries, that’s not rude at all; we wind up asking the same sort of questions many times a day.

#18 Updated by Markus Roberts over 5 years ago

  • Status changed from Investigating to Needs More Information
  • Branch set to http://github.com/MarkusQ/puppet/tree/ticket/2.6.x/4409

I have not been able to reproduce this (it seems to search the right directories when I try it) but I was getting some warnings and have a branch which fixes those (other than a deprecation notice on changing from foo/plugins to foo/lib) and adds some diagnostics. It’s at

http://github.com/MarkusQ/puppet/tree/ticket/2.6.x/4409

Could someone who is seeing this try with that branch and report back the messages from the master?

#19 Updated by Rudy Gevaert over 5 years ago

Dan, I would really appreciate it if you could test that. As you were able to reproduce. I will probably have some ‘free’ time, the earliest, on Friday. And I would still need to try to get to run puppet from that source. I’m running the it with Debian packages. And I don’t it to interfere with the current setup.

Please Dan :)

#20 Updated by James Turnbull over 5 years ago

Rudy – I doubt Dan is available as I believe he is in Europe.

#21 Updated by Nigel Kersten over 5 years ago

Markus, I can confirm that a checkout of your branch is working for me.

root@nigelk-vm1:~# puppetmasterd --version
2.6.0

root@nigelk-vm1:~# puppetmasterd --config /etc/puppet/puppetmasterd.conf --configprint libdir
/var/lib/puppetmaster/lib

root@nigelk-vm1:~# ls -l /var/lib/puppetmaster/lib/
total 0

root@nigelk-vm1:~# grep -r newtype /var/lib/puppetmaster/environments/test_environment/modules/base/lib/puppet
/var/lib/puppetmaster/environments/test_environment/modules/base/lib/puppet/type/filekeyvalue.rb:  newtype(:FileKeyValue) do

root@nigelk-vm1:~# grep -i filekeyvalue /var/lib/puppetmaster/environments/test_environment/modules/base/manifests/init.pp 
fileKeyValue { "enable_gssd":

Client runs are working happily against this.

info: Applying configuration version '1283531294'
notice: /Stage[main]/Base/Filekeyvalue[enable_gssd]/ensure: ensure changed 'outofsync' to 'insync'
notice: Finished catalog run in 0.53 seconds

That was supposed to show that I have an empty libdir as far as puppetmasterd is concerned, that I’m distributing a custom type in modules/base/lib/puppet, that this type is employed in the base module, and the client run works.

My only caveat is that this was tested with a 0.25.4 client.

#22 Updated by Nigel Kersten over 5 years ago

Markus, did you still need additional debug info from the master if the discovery of the custom type was successful?

#23 Updated by Markus Roberts over 5 years ago

Nigel —

I’m unsure. My branch just dealt with a warning which should not have led to the reported failure, and did not for me, running 2.6,x both client & server, and added some diagnostic message. Since I was unable to reproduce the problem I was hoping that it would fail with my branch and provide more information to help track down the problem.

A few thoughts:

  • If you try the same test you ran this morning with 2.6.0, do you see the reported problem?
  • I’m thinking the problem may just have been misdiagnosed in comment 7, and (if no one else can reproduce it) the best next step may be to go back to the original vcrepo specific problem rather than the generalization.

#24 Updated by Nigel Kersten over 5 years ago

Yyou are correct, this was misdiagnosed somehow, and my misconceptions haven’t helped here, as I thought this was an issue throughout 0.25.x

2.6.0 does not have this problem.

0.25.5 does not have this problem.

0.25.4 does

info: Caching node for 9611c15a-c143-41ba-9c9f-10b96588a727
info: Autoloaded module base
err: Could not find resource type filekeyvalue at /var/lib/puppetmaster/environments/test_environment/modules/base/manifests/init.pp:29 on node 9611c15a-c143-41ba-9c9f-10b96588a727
err: Could not find resource type filekeyvalue at /var/lib/puppetmaster/environments/test_environment/modules/base/manifests/init.pp:29 on node 9611c15a-c143-41ba-9c9f-10b96588a727

Copying modules/base/lib/puppet to /var/lib/puppetmaster/lib/puppet resolves this error.

No mention in the release notes, so perhaps it was fixed by accident?

#25 Updated by Rudy Gevaert over 5 years ago

Hi Markus and Nigel. I managed to get your branch working here on a test machine. (It took me more time to figure out how to checkout the branch than to get it to work…)

Running master and agent (same version) on the same machine works perfect.

However I then setup an other machine and ran the agent on it. (from the branch).

Please see the attached file for the output. Bottom line:

Could not run Puppet configuration client: Could not find a default provider for vcsrepo

On the agent I also see:

pupclient:~# tree /var/lib/puppet/lib/
/var/lib/puppet/lib/
|-- facter
|   `-- configured_ntp_servers.rb
`-- puppet
    |-- provider
    |   |-- a2mod
    |   |   `-- a2mod.rb
    |   |-- vcsrepo
    |   |   |-- bzr.rb
    |   |   |-- cvs.rb
    |   |   |-- git.rb
    |   |   |-- hg.rb
    |   |   `-- svn.rb
    |   `-- vcsrepo.rb
    `-- type
        |-- a2mod.rb
        `-- vcsrepo.rb

6 directories, 10 files
pupclient:~# 

#26 Updated by Nigel Kersten over 5 years ago

Rudy, can you try a different plugin so we can see if it’s a problem with this particular vcsrepo or whether you have something more general going on?

Does your a2mod type/provider work if you comment out your vcsrepo resource?

#27 Updated by Rudy Gevaert over 5 years ago

Hi Nigel. I was able to get the puppetlabs apache module to work (with some minor adjustments to the code). Trying the vcsrepo module again gives

err: Could not run Puppet configuration client: Could not find a default provider for vcsrepo

#28 Updated by Markus Roberts over 5 years ago

Could this be a run-twice problem: run once to sync the provider, so that it will be available for use on the second run? Or could it be, as Nigel suggested, something specific to the vcsrepo?

#29 Updated by James Turnbull over 5 years ago

The Apache module is:

http://forge.puppetlabs.com/puppetlabs/apache

It’s Github is:

http://github.com/puppetlabs/puppetlabs-apache

#30 Updated by James Turnbull over 5 years ago

  • Target version changed from 2.6.1 to 2.7.x

So I think we’ve established that plugins work in environments but in our testing we’ve struck the “needs two runs” issues with custom plugins. That issue won’t be resolved for 2.6.1

#31 Updated by James Turnbull over 5 years ago

  • Status changed from Needs More Information to Closed
  • Target version changed from 2.7.x to 2.6.1

#32 Updated by Rudy Gevaert over 5 years ago

When 2.6.2 is released I’m going to have a look at this issue again then :) Thanks for the help!

#33 Updated by John Warburton over 5 years ago

I am having this problem too.

I replicated the setup like Nigel’s 0.25.4 above but couldn’t seem to get it to work. Am I missing a configuration setting?

My client connects in the ‘prod’ environment:

`-- prod
    |-- manifests
    |   `-- site.pp
    `-- modules
        `-- testmodule
            |-- lib
            |   |-- facter
            |   |   `-- nothing.rb
            |   `-- puppet
            |       `-- type
            |           `-- logadm.rb
            `-- manifests
                `-- init.pp

But can’t see the logadm type unless I put it in $libdir/puppet/type

I just get

err: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type logadm at /tmp/puppet26/environments/prod/modules/testmodule/manifests/init.pp:15 on node corwadm010.bfm.com

I have included the full configuration of client & server, plus the debugging information generated from Markus’s branch in comment #18

#34 Updated by Nigel Kersten over 5 years ago

  • Status changed from Closed to Investigating
  • Assignee changed from Markus Roberts to Nigel Kersten
  • Target version changed from 2.6.1 to 69

John, after I posted my successful runs above, I spent some time experimenting and didn’t capture the results in this ticket.

I found that running Passenger/Apache, it always worked for me.

However, when I was running webrick, it would sometimes fail to find the plugins. I never managed to work out what the actual problem was, as it would seemingly start working for no reason at all, and break for no reason at all.

Were you testing with webrick and a plain ‘puppetmasterd’ invocation?

#35 Updated by John Warburton over 5 years ago

Nigel

I found this out when running behind passenger. My stack is:

  • puppet (2.6.3)
  • ruby (1.8.7-p249)
  • apache (2.2.15)
  • fastthread (1.0.7)
  • passenger (2.2.14)
  • rack (1.1.0)
  • rake (0.8.7)

I get the same errors. NB I have removed the local cache:

root@263client# /opt/local/sbin/run-puppet.sh --color true
+ /opt/local/sbin/puppetd --server puppet.bfm.com --verbose --onetime --no-daemonize --no-usecacheonfailure --config /var/puppet/etc/puppetd.conf --environment prod --color true
info: Retrieving plugin
notice: /File[/var/puppet/var/lib/facter]/ensure: created
notice: /File[/var/puppet/var/lib/facter/cyberark_init.rb]/ensure: defined content as '{md5}fef248a67ea8794c0dfc022906af94cb'
notice: /File[/var/puppet/var/lib/facter/pkgs_facts.rb]/ensure: defined content as '{md5}272135f0a27322df0721fb1519d5a59e'
notice: /File[/var/puppet/var/lib/facter/serialnumber.rb]/ensure: defined content as '{md5}e1fdafc8a460867f7f6d665ad9280202'
notice: /File[/var/puppet/var/lib/facter/solaris_memory.rb]/ensure: defined content as '{md5}07485f55c10e14710e50465859971ec9'
notice: /File[/var/puppet/var/lib/facter/svcs_facts.rb]/ensure: defined content as '{md5}6325f65a1948372902e3a7c818697de2'
notice: /File[/var/puppet/var/lib/puppet]/ensure: created
notice: /File[/var/puppet/var/lib/puppet/parser]/ensure: created
notice: /File[/var/puppet/var/lib/puppet/parser/functions]/ensure: created
notice: /File[/var/puppet/var/lib/puppet/provider]/ensure: created
notice: /File[/var/puppet/var/lib/puppet/type]/ensure: created
notice: /File[/var/puppet/var/lib/puppet/type/logadm.rb]/ensure: defined content as '{md5}21e554a8a34dcdea2f2671380d30df24'
info: Loading downloaded plugin /var/puppet/var/lib/puppet/type/logadm.rb
info: Loading downloaded plugin /var/puppet/var/lib/facter/solaris_memory.rb
info: Loading downloaded plugin /var/puppet/var/lib/facter/pkgs_facts.rb
info: Loading downloaded plugin /var/puppet/var/lib/facter/svcs_facts.rb
info: Loading downloaded plugin /var/puppet/var/lib/facter/serialnumber.rb
info: Loading downloaded plugin /var/puppet/var/lib/facter/cyberark_init.rb
info: Loading facts in pkgs_facts
info: Loading facts in cyberark_init
info: Loading facts in solaris_memory
info: Loading facts in serialnumber
info: Loading facts in svcs_facts
info: Loading facts in pkgs_facts
info: Loading facts in cyberark_init
info: Loading facts in solaris_memory
info: Loading facts in serialnumber
info: Loading facts in svcs_facts
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type logadm at /local/file-repo/prod/modules/apache_infra_server/manifests/init.pp:120 on node corwadm010.bfm.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

This is weird – I forgot to mention this works absolutely fine in 0.25.5 with the same stack & same method of generating the config files, which is how I found the issue

root@0255client# /opt/local/sbin/run-puppet.sh --tags puppet_client
+ /opt/local/sbin/puppetd --server puppet-lab.bfm.com --verbose --onetime --no-daemonize --ignorecache --no-usecacheonfailure --config /var/puppet/confdir/puppetd.conf --environment lab --tags puppet_client
info: Retrieving plugin
notice: /File[/var/puppet/confdir/var/lib/puppet/type/logadm.rb]/ensure: content changed '{md5}21e554a8a34dcdea2f2671380d30df24' to '{md5}21e554a8a34dcdea2f2671380d30df24'
info: Loading downloaded plugin /var/puppet/confdir/var/lib/puppet/type/logadm.rb
info: Loading facts in pkgs_facts
info: Loading facts in serialnumber
info: Loading facts in cyberark_init
info: Loading facts in solaris_memory
info: Loading facts in svcs_facts
info: Loading facts in pkgs_facts
info: Loading facts in serialnumber
info: Loading facts in cyberark_init
info: Loading facts in solaris_memory
info: Loading facts in svcs_facts
info: Caching catalog for engncfm001.bfm.com
info: Applying configuration version '1290119477'
notice: //puppet_client/Logadm[/var/log/puppet_client/puppet_client.log]/ensure: created
notice: Finished catalog run in 11.60 seconds

Could the logadm code be tickling a 2.6 edge case here? Logadm_Patterns

#36 Updated by Nigel Kersten over 5 years ago

  • Status changed from Investigating to Accepted
  • Assignee deleted (Nigel Kersten)

#37 Updated by James Turnbull over 5 years ago

  • Target version changed from 69 to 2.6.5

#38 Updated by Nigel Kersten over 5 years ago

  • Status changed from Accepted to Needs More Information
  • Target version changed from 2.6.5 to 69

What happens if you remove the logadm type/provider? Do things work happily then?

#39 Updated by Nigel Kersten over 5 years ago

  • Target version changed from 69 to 2.6.5

#40 Updated by John Warburton over 5 years ago

  • Target version changed from 2.6.5 to 69

If I remove the provider from $libdir/puppet/type, and ensure that a change needs to be made, the provider will fail as above.

Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type logadm at /local/file-repo/prod_blk/modules/puppet_client/manifests/init.pp:95 on node cornadm010.example.com

#41 Updated by John Warburton over 5 years ago

ugh – somehow the target version got changed back to 2.6.x

#42 Updated by Nigel Kersten over 5 years ago

  • Target version changed from 69 to 2.6.5

#43 Updated by Nigel Kersten over 5 years ago

Can you comment out all the logadm resources as well?

We’ve had some issues with that, it wasn’t ever committed into the core, and it looks to have been broken in any case in 2.6.x

#44 Updated by John Warburton over 5 years ago

When I comment out the logadm resources, or no changes need to made, then there is no error.

It is only when I need to use the logadm resource that it needs to be in $libdir/puppet/type rather than $modulepath/modulename/lib/puppet/type

NB I still have logadm.rb in $modulepath/modulename/lib/puppet/type

#45 Updated by Nigel Kersten over 5 years ago

do you have any other custom types/plugins you’re distributing?

I’m trying to work out whether or not this specific plugin is the problem, or whether there is still a general problem with pluginsync.

I’m asking because in my last job I did spend an awful lot of time debugging what appears to be the same problem, and simply couldn’t ever reproduce this issue with my own types/plugins with Passenger.

#46 Updated by John Warburton over 5 years ago

We do not have any other types, so I copied across the simple example from http://www.kartar.net/2010/02/puppet-types-and-providers-are-easy/

I see the same behaviour as before

I put the provider and type in the same “common” module as logadm:

% find modules/common/lib/puppet -name "*.rb"
modules/common/lib/puppet/provider/repo/svn.rb
modules/common/lib/puppet/provider/repo/git.rb
modules/common/lib/puppet/type/logadm.rb
modules/common/lib/puppet/type/repo.rb

The new providers & type copy across fine, but I get the same error as before, when using the example resource

repo { "sandbox":
    source => "https://puppet-svn.example.com/svn-sandbox/warbjoh",
    path => "/var/tmp/sandbox",
    ensure => present,
}

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type repo at /u1/warbjoh/svn-workspace/puppet/trunk/modules/subversion_client/manifests/init.pp:40 on node warbjohn.example.com

If I put the providers and type into $libdir/puppet/provider & $libdir/puppet/type, the resource executes, getting the svn errors I expect from a first time svn checkout:

err: /Stage[main]/Subversion_client/Repo[sanbox]/ensure: change from absent to present failed: Execution of '/opt/local/bin/svn checkout https://puppet-svn.example.com/svn-sandbox/warbjoh /var/tmp/sandbox' returned 1: Error validating server certificate for 'https://puppet-svn.example.com:443':

#47 Updated by Nigel Kersten over 5 years ago

  • Status changed from Needs More Information to Accepted
  • Priority changed from High to Urgent

Thank you. That’s much appreciated.

#48 Updated by Alan Barrett over 5 years ago

I don’t think this is a bug.

The provider needs to be installed on both the master and the client before it will work. The pluginsync stuff will install it on the client, but you haven’t done anything to install it on the master.

Expecting the master to auto-load code using the modulepath seems wrong to me; the modulepath is where the master searches for stuff to compile and/or send to the client, not where the master searches for stuff to execute itself. If you do want to use the modulepath to load code to execute on the master, then how would you deal with different versions of the code being present in different instances of the modulepath as used for different environments?

#49 Updated by Nigel Kersten over 5 years ago

It is a bug Alan. You should be able to distribute a type/provider pair simply by placing it into a module.

Otherwise it’s impossible to maintain strict release control over different versions of the type/provider in different environments that may accept different parameters.

It only needs to be “on the master” to validate the manifests, so you really only need the type visible to the server, the provider doesn’t actually need to do any work on the master.

This was fixed in 0.25.x, and it appears we’ve broken it again in 2.6.x.

#50 Updated by Alan Barrett over 5 years ago

This was fixed in 0.25.x, and it appears we’ve broken it again in 2.6.x.

OK. My expectations are derived partially from experience with 0.24.

So is there supposed to be some magic in which the master uses different copies of the code for the type and/or provider depending on which client it is talking to? I know the provider doesn’t actually do much on the master, but at least things like has_feature need to work.

#51 Updated by Nigel Kersten over 5 years ago

has_feature shouldn’t come into it though.

The master just validates the type as used in the manifests, the client does the actual choosing of a provider to be paired with the specified type which is where the features/default provider/commands/etc all come into play.

#52 Updated by Paul Berry over 5 years ago

Jesse Wolfe and I have spent some time investigating this bug. The key problem is that a master can’t reliably distinguish between two versions of the same native type that are used in different environments (or between an environment which uses a native type and an environment which doesn’t). This is due to the fact that native types are defined using ruby code, which the master loads into Ruby classes via “require”. Since there can only be one Ruby class with a given name at a time, this prevents the master from being able to have two different versions of the same type in two different environments. This makes life difficult for people who are trying to use a “test” environment to try out a new version of a native type on a limited set of nodes before deploying it to all nodes.

A secondary problem is that the location where the master looks for the definition of native types is not the same as the location of plug-in files that the master distributes to agents. This leads to confusion even for people who are not using a “test” environment, because it means that they have to put their type definitions in two places, one where they can be picked up by the master and one where they can be sent as plug-ins to agents.

In the short term, we recommend addressing both problems by moving the responsibility for validating resources from the master to the agent, so that it isn’t necessary for the master to be aware of type definitions at all. This carries a small disadvantage, which is that if a user makes an error in their manifest that is caught by validation, the agent will already have received the new, invalid catalog, so it won’t be able to fall back to the old catalog. This feels like a small price to pay for fixing this bug.

In the long term, we’d like to change the way a native type is defined so that instead of using native ruby code, it is simply a piece of language-independent metadata (e.g. JSON). This would allow the master to validate resources again, since it would be able to keep track of the definitions of types on a per-environment basis and load the definition of each type from its environment-specific directory.

#53 Updated by Peter Meier over 5 years ago

In the short term, we recommend addressing both problems by moving the responsibility for validating resources from the master to the agent, so that it isn’t necessary for the master to be aware of type definitions at all. This carries a small disadvantage, which is that if a user makes an error in their manifest that is caught by validation, the agent will already have received the new, invalid catalog, so it won’t be able to fall back to the old catalog. This feels like a small price to pay for fixing this bug.

Couldn’t we address that by caching the catalog on the agent after validation. Assuming that validation is a separate process run before trying to apply the catalog.

#54 Updated by Nigel Kersten over 5 years ago

  • Target version changed from 2.6.5 to 2.7.x

I’m re-targeting this at Statler as it looks like we don’t have a good viable solution for 2.6.x.

#55 Updated by Yuri Arabadji about 5 years ago

You know what, I came across similar problem recently. This time not only I’m using custom types & providers, but exported resources, too.

Here’s how the error looks like in 2.6.7:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Resource type zenoss_host doesn't exist at /etc/puppet/development/modules/zenoss/manifests/init.pp:47 on node xxx

And in master’s log:

puppet-master[1639]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type zenoss_host at /etc/puppet/development/modules/zenoss/manifests/init.pp:38 on
node xxx

Master’s lib is empty:

# tree  /var/lib/puppet/lib/
/var/lib/puppet/lib/
0 directories, 0 files

Here’s the proposed solution:

  • maintain $libdir/environments/$environment directory that corresponds to client’s environment
  • implement “resync on each manifest compile” feature that scans $modulepath for providers & types and sync’s the files to the dir above
  • create an option in config that allows toggling of “resync on each run” feature — so that user can set it to “on” when developing and “off” when in production.

Current band-aid is to invoke this on each provider/type change:

rsync -av --exclude .svn --exclude .git  /etc/puppet/development/modules/*/lib/puppet/{type,provider} /var/lib/puppet/lib/puppet/

#56 Updated by Stijn Hoop almost 5 years ago

Running into this as well, which was frustrating because all I did was combine the chapter 3 and 10 of the new (and mostly excellent) Pro Puppet book. Due to the very confusing error message it took me a while to connect to this bug.

Note that while the workaround of rsyncing / symlinking the modules into the servers modulepath works, it is still necessary to restart the puppetmaster every time that module changes. That means that practically speaking I cannot really recommend putting new types/providers in modules for 2.6.x, at least when also using environments. I hope it will be fixed in 2.7.x…

#57 Updated by eric sorenson over 4 years ago

Checking in — Is this changed at all in 2.7.9 and newer?

#58 Updated by James Turnbull over 4 years ago

  • Status changed from Accepted to Needs Decision
  • Assignee set to Jason McKerr

Jason – can we throw some eyes on this and do an update please.

#59 Updated by Jason McKerr over 4 years ago

  • Assignee changed from Jason McKerr to Anonymous

#60 Updated by Anonymous over 4 years ago

James Turnbull wrote:

Jason – can we throw some eyes on this and do an update please.

Hey. So, we checked this, and it looks like this has mostly been a lack-of-communication problem. This is still on the road map for the open source team, found here: http://projects.puppetlabs.com/projects/puppet/wiki/Road_map

We have some idea of the change to fix this at a design level, but the scope of it is large enough that we don’t feel comfortable attacking it in the 2.7 series at all. (It involves fundamental changes to the way we do type class lookup, how we interact with the metaprogramming around them, and a whole bunch of other very fundamental infrastructure inside the compiler and the agent.)

Hopefully this should be delivered in the next major release of Puppet, but I don’t have a concrete schedule right now. We are aware that there is substantial priority behind this from a usability point of view for people using environments, and it is mostly blocked by other, easier to hit problems in the same space.

#61 Updated by James Turnbull over 4 years ago

  • Status changed from Needs Decision to Accepted
  • Target version changed from 2.7.x to 3.x

#62 Updated by Nigel Kersten over 4 years ago

This is one of our must-delivers for Telly. It’s a more complex problem than it may appear.

#63 Updated by Nigel Kersten over 4 years ago

  • Status changed from Accepted to Closed

The original ticket was around plugins in modules not working anymore in 2.6.0 when used in environments.

That problem has been solved, and thus I’m closing this ticket.

We diverged discussion into a different issue, which I’ve characterized in the following ticket:

#12173 – Masters cannot reliably distinguish between multiple versions of a type/function/plugin used in different environments

Please watch that ticket if you care about this specific issue.

#64 Updated by Thomas Vander Stichele about 4 years ago

Nigel, you ay the problem of plugins in modules not working anymore in 2.6.0 has been fixed? Which comment leads you to believe that? I am running 2.6.12 and I’m seeing the same issue.

When I apply my development environment it is correctly finding a new firewallchain type from a branch of puppetlabs-firewall. When I apply the production environment (with the modules directory with the exact same contents) I get

Could not evaluate: Could not retrieve information from environment production source(s) puppet://puppetmaster/plugins

I quite like the approach proposed in http://projects.puppetlabs.com/issues/4409#note-55 –> separate plugin directories per environment. You should probably copy or reference that note in #12173 because it seems the only straightforward workaround for this bug.

In any case, I don’t have the perception this bug is closed, and this pretty much makes environments unusable for any serious puppet work, since the purpose of environments is to test code (including modules and plugins) in a different environment than production.

#65 Updated by Nigel Kersten about 4 years ago

I was running with plugins in modules in large production in 2.6.x, but not with an environment called “production”.

Can you do a test with a different named environment with the same modules and see if you hit this problem?

#66 Updated by Nigel Kersten about 4 years ago

I should also add that 2.6.x is only getting security updates right now, and bug fixes are targeted at 2.7.x.

#67 Updated by Thomas Vander Stichele about 4 years ago

  • Target version changed from 3.x to 2.7.x

Nigel Kersten wrote:

I was running with plugins in modules in large production in 2.6.x, but not with an environment called “production”.

Can you do a test with a different named environment with the same modules and see if you hit this problem?

Not sure I follow. Isn’t ‘production’ the default environment, the one you get when you don’t specify any environment ?

#68 Updated by Nigel Kersten about 4 years ago

Yes, but I didn’t use it at all. Every node had an explicitly mapped environment, and no environments were called ‘production’, so there may be an issue in that area.

#69 Updated by Anonymous about 4 years ago

  • Status changed from Closed to Needs More Information

Nigel, this was closed, and had activity. What state should it be in?

#70 Updated by Nigel Kersten about 4 years ago

Daniel, it does look like we still have an issue here.

I suggest we make a new ticket with a fresh description, and mark this one as a duplicate of that.

That way we avoid historical cruft, have a clean description of the problem as it applies to 2.7.x, and the watchers get transferred over.

I don’t know where we regressed here, and I’m starting to question my memory. I should point out though that 2.6.x is in ‘security fix only’ mode.

#71 Updated by eric sorenson about 4 years ago

Nigel Kersten wrote:

I suggest we make a new ticket with a fresh description, and mark this one as a duplicate of that.

I’ve tried to succinctly capture the issue as it applies to current versions in #13858. Please let me know if I’ve gotten anything wrong there; if not it would be a candidate for a “forward dupe” here.

#72 Updated by Sandor Sklar about 4 years ago

I will be out of the office and reading email sporadically through Monday, April 9th, and will respond to your email then.

For critical issues or systems emergencies, please contact Kam Chan, Rob Smith, Julian Morley, and/or Tom Cramer.

Thanks,

-- Sandy

#73 Updated by Anonymous about 4 years ago

  • Status changed from Needs More Information to Closed

Anyone who continues to care about this should follow the discussion in #13858.

#74 Updated by Anonymous over 3 years ago

  • Target version deleted (2.7.x)

#75 Updated by Charlie Sharpsteen about 3 years ago

  • Keywords set to customer

#77 Updated by Sandor Sklar about 3 years ago

  • Support Urls deleted (http://this.is.a.customer.support.ticket/Apple http://nokia.non.paying.user/cares/about/this)

I will be out of the office through Monday, April 8th, and will be responding to email sporadically through that date.

— Sandy

#78 Updated by Anonymous almost 3 years ago

  • Assignee deleted (Anonymous)

Also available in: Atom PDF