The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #17010

win32-dir gem returns a string whose encoding is not `ascii_compatible?`

Added by Ed Sumerfield over 3 years ago. Updated over 2 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Josh Cooper% Done:

0%

Category:-
Target version:3.2.4
Affected Puppet version:3.0.0 Branch:
Keywords:windows

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Problem originally defined by this exception:

: irb
irb(main):001:0> require "rspec-puppet"
Failed to load feature test for root: uninitialized constant Windows::Synchronize
ArgumentError: string contains null byte
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/util/run_mode.rb:67:in `join'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/util/run_mode.rb:67:in `conf_dir'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/settings.rb:495:in `user_config_file'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/settings.rb:1234:in `which_configuration_file'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/settings.rb:475:in `parse_config_files'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet/settings.rb:147:in `initialize_global_settings'

        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet.rb:135:in `do_initialize_settings_for_run_mode'

        from C:/ruby193/lib/ruby/gems/1.9.1/gems/puppet-3.0.0/lib/puppet.rb:123:in `initialize_settings'
        from C:/ruby193/lib/ruby/gems/1.9.1/gems/rspec-puppet-0.1.5/lib/rspec-puppet.rb:10:in `<top (required)>'
        from C:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:60:in `require'
        from C:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:60:in `rescue in require'
        from C:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
        from (irb):1
        from C:/ruby193/bin/irb:12:in `<main>'

Research concluded that ruby 1.9.3 does not support File.join of strings that are not UTF-8. There is a ruby fix associated with this problem mentioned here:

https://bugs.ruby-lang.org/issues/7168

Since this fix is not useful for 1.9.3 the following patch is a possible way around it.

class File
  class << self
    alias_method :original_join, :join
  end

  def self.join(*args)
    new_args = args.collect { |questionableEncoding|
      join_encoding_fix(questionableEncoding)
    }
    self.send(:original_join, new_args)
  end

  def self.join_encoding_fix(value)
    if (value.instance_of?(String))
      value = value.encode("UTF-8")
    elsif (value.instance_of?(Array))
      value = value.collect { |subValue|
        join_encoding_fix(subValue)
      }
    end
    value
  end
end

Related issues

Related to Facter - Bug #21285: Facter.processorcount fails on Windows 64bit Merged - Pending Release
Related to Puppet - Bug #20522: Improve Puppet's handling of non-ASCII character encodings Accepted
Related to Puppet - Bug #22493: Can't start puppet agent on non english Windows Closed

History

#1 Updated by Anonymous over 3 years ago

If I understand the upstream bug correctly, all File paths must be encoded using an encoding that is compatible with ASCII. UTF8 is compatible with ASCII for example, so this works:

irb(main):003:0> File.join("a".encode("UTF-8"), "".encode("UTF-8"))
=> "a/"

But UTF-16 is mutually exclusive with ASCII, hence the error:

irb(main):004:0>  File.join("a".encode("UTF-16LE"), "".encode("UTF-16LE"))
ArgumentError: string contains null byte
        from (irb):4:in `join'
        from (irb):4
        from /Users/jeff/.rbenv/versions/1.9.3-p194/bin/irb:12:in `
'

POSIX paths are pretty clear about null bytes, so this all seems to make sense and is consistent.

What we do about it in Puppet though… =)

-Jeff

#2 Updated by Josh Cooper over 3 years ago

  • Description updated (diff)
  • Status changed from Unreviewed to Accepted

#3 Updated by Josh Cooper over 3 years ago

  • Target version set to 3.0.x
  • Affected Puppet version set to 3.0.0

I believe this is a bug in the win32/dir gem in that it should be returning a string whose encoding is Encoding.ascii_compatible? rather than UTF-16LE.

From https://groups.google.com/d/topic/ruby-core-google/aiW7jRNWRvk/discussion:

|> Does anyone have information as to the current status of
|> adding Unicode-savvy path handling to 1.9 ruby?
|
|Ugh.  Sorry, I mean of course: Unicode-savvy path handling
|on *win32* ruby 1.9.
Every path encoding is UTF-8 and converted to UTF-16 internally.  If
there's something still use *A functions, it will eventually replaced
by *W functions.  In short, if you're using UTF-8 for your program
encoding, you should not see any problem  (if you do, it's a bug).

                                                        matz.

The ruby 1.9.3 source also has native code for resolving windows special folders. It calls the *W version of SHGetSpecialFolderLocation:

static BOOL
get_special_folder(int n, WCHAR *env)
{
    LPITEMIDLIST pidl;
    LPMALLOC alloc;
    BOOL f = FALSE;
    if (SHGetSpecialFolderLocation(NULL, n, &pidl) == 0) {
      ...

And converts the wchar_t string using the current filesystem encoding:

VALUE
rb_w32_special_folder(int type)
{
    WCHAR path[_MAX_PATH];

    if (!get_special_folder(type, path)) return Qnil;
    regulate_path(path);
    return rb_w32_conv_from_wchar(path, rb_filesystem_encoding());
}

Okay, it turns out we can eliminate the need for win32/dir in ruby 1.9.3:

irb(main):023:0> require 'etc'
irb(main):026:0> Etc.sysconfdir
=> "C:/ProgramData"
irb(main):027:0> Etc.sysconfdir.encoding
=> #<Encoding:Windows-1252>

But those methods are not available in ruby 1.8.7, so we’ll need to fall back to win32/dir

C:\work\puppet>irb
irb(main):001:0> require 'etc'
=> false
irb(main):002:0> Etc.sysconfdir
NoMethodError: undefined method `sysconfdir' for Etc:Module
        from (irb):2

#4 Updated by Josh Cooper over 3 years ago

  • Subject changed from Puppet 3.0 - Windows - Char Encoding Path Problem to win32-dir gem returns a string whose encoding is not `ascii_compatible?`

#5 Updated by Anonymous over 3 years ago

  • Status changed from Accepted to Needs More Information
  • Assignee set to Josh Cooper

Since this is a bug in the win32/dir gem, what can we do about it? Are there a few places in the code where we can re-encode things?

#6 Updated by Josh Cooper over 3 years ago

  • Keywords set to windows

If we move to ruby19, we can eliminate the usage of that gem. In the meantime, we can submit a pull request upstream and/or patch the few places we use it, e.g. install.rb, run_mode

#7 Updated by Josh Cooper over 3 years ago

  • Status changed from Needs More Information to Accepted

#8 Updated by Anonymous over 3 years ago

  • Target version changed from 3.0.x to 3.x

As the 3.0.x line is winding down with the impending release of 3.1.0, I am removing the target at 3.0.x from tickets in the system and targeting them at 3.x instead.

#9 Updated by Josh Cooper about 3 years ago

  • Status changed from Accepted to Closed

I can no longer reproduce this problem:

C:\ruby193\bin>ruby --version
ruby 1.9.3p392 (2013-02-22) [i386-mingw32]
C:\ruby193\bin>irb
irb(main):001:0> require 'win32/dir'
=> true
irb(main):002:0> Dir::WINDOWS.encoding
=> #<Encoding:US-ASCII>
irb(main):003:0> Dir::WINDOWS.encoding.ascii_compatible?
=> true
irb(main):004:0> require 'rspec-puppet'
=> true
C:\>gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
columnize (0.3.6)
debugger-linecache (1.2.0)
debugger-ruby_core_source (1.2.0)
diff-lcs (1.2.3)
ffi (1.7.0 x86-mingw32)
io-console (0.3)
json (1.5.5)
metaclass (0.0.1)
minitar (0.5.4)
minitest (2.5.1)
mocha (0.13.3)
puppetlabs_spec_helper (0.4.1)
rake (0.9.2.2)
rdoc (3.9.5)
rgen (0.6.2)
rspec (2.13.0)
rspec-core (2.13.1)
rspec-expectations (2.13.0)
rspec-mocks (2.13.1)
rspec-puppet (0.1.6)
stomp (1.2.9)
sys-admin (1.5.6 x86-mingw32)
win32-api (1.4.8 x86-mingw32)
win32-dir (0.3.7)
win32-eventlog (0.5.3)
win32-process (0.6.5)
win32-security (0.1.4)
win32-service (0.7.2 x86-mingw32)
win32-taskscheduler (0.2.2)
win32console (1.3.2 x86-mingw32)
windows-api (0.4.2)
windows-pr (1.2.2)

Ed, can you try the latest 3.2 RC1? If the issue still persists, please reopen the ticket.

#10 Updated by Josh Cooper almost 3 years ago

  • Status changed from Closed to Accepted

This is reproducible on non en-US locales. See https://projects.puppetlabs.com/issues/21285#note-5.

#11 Updated by Josh Cooper almost 3 years ago

  • Status changed from Accepted to Merged - Pending Release
  • Target version changed from 3.x to 3.2.4

This will be fixed in 3.2.4 as the win32-dir gem has been updated to 0.4.3. Alternatively, you can do a gem update win32-dir.

See also #21285 (same bug in facter)

#12 Updated by Josh Cooper over 2 years ago

  • Status changed from Merged - Pending Release to Closed

Also available in: Atom PDF