Bug #10234

failed puppet apply makes delayed_job workers crash and disk fills up with yaml files

Added by Evgeny Zislis 7 months ago. Updated 6 months ago.

Status:Accepted Start date:10/22/2011
Priority:Normal Due date:
Assignee:- % Done:

0%
Category:-
Target version:2.7.x
Affected Puppet version:2.7.1 Branch:
Keywords:
Votes: 0

Description

A failed puppet apply has unicode (or something) in the message, this makes delayed_job die when trying to process the yaml file in puppet dashboard.

traceback.txt - delayed_job traceback (12.6 kB) Evgeny Zislis, 10/22/2011 11:36 pm

failed-report.yaml - yaml file with failed report (1 kB) Evgeny Zislis, 10/22/2011 11:36 pm

History

#1
Updated by Josh Cooper 6 months ago

  • Status changed from Unreviewed to Needs More Information

Hi Evgeny,

The puppet master is failing to load the node yaml file for host116.xxx. Can you attach $vardir/yaml/node/host116.xxx from the puppet master to this issue (after removing any sensitive information)?

#2
Updated by Evgeny Zislis 6 months ago

I did attach a failed report.

#3
Updated by Josh Cooper 6 months ago

Hi Evgeny,

Sorry, I should have been more specific. The real error is occurring on the server, when trying to load the file $vardir/yaml/node/host116.xxx. The server then reports an error to the client, which then writes the error in the report, which you attached. But I need to see the original yaml file on the server.

#4
Updated by Evgeny Zislis 6 months ago

Look at the attached files. The yaml report is there.

#5
Updated by Josh Cooper 6 months ago

Hi Evgeny,

I’m actually not interested in the failed-report.yaml you attached. I’m interested in the file that the server failed to parse. It should be located at $vardir/yaml/node/.yaml.

The file should contains facts that the node uploaded to the server. Likely there is some fact that contains non-latin characters, and something along the way is not encoding them correctly.

The fact that the data becomes part of an error message in the failed-report.yaml, which later causes problems for dashboard, is secondary.

The file I’d like you to attach should contain information like the following:

--- !ruby/object:Puppet::Node
  classes: []
  environment: &id004 production
  expiration: 2011-11-21 12:38:36.666265 -08:00
  name: foo1
  parameters: 
    sp_boot_rom_version: MBP61.0057.B0C
    sp_number_processors: "2"
...

#6
Updated by Evgeny Zislis 6 months ago

This bug is about the delayed job workers dying. I can’t provide the compiled catalog file at this time, but the report containing Unicode characters (which is causing this problem) could appear for any number of reasons. Please fix the report parsing in the dashboard so workers live longer. I’ll create a new ticket for the error you are asking, next week, when I’m in the office.

#7
Updated by Josh Cooper 6 months ago

  • Status changed from Needs More Information to Accepted
  • Target version set to 2.7.x

Hi Evgeny,

Yeah, I understand. It’s just that the report contains either raw unicode U-C08D U-3D0C or it contains an illegal UTF-8 sequence, and having the original facts would help debug. But I agree that the delayed job should not die when encountering parse errors like this.

Also available in: Atom PDF