Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(maint) Avoid deadlock of Facter::Core::Execution.execute #2114

Merged

Conversation

oanatmaria
Copy link
Contributor

@oanatmaria oanatmaria commented Oct 1, 2020

Description of the problem: If stdout/stderr are N bytes, and the child process writes N+1 bytes to stderr, but nothing to stdout, then facter will block waiting to read from stdout, hit the timeout, and never read from stderr.
Description of the fix Facter will create two reader threads to avoid the deadlock

@oanatmaria oanatmaria requested review from a team October 1, 2020 11:25
@oanatmaria oanatmaria added the bug Something isn't working label Oct 1, 2020
@oanatmaria oanatmaria force-pushed the long_stderr_during_execution branch 4 times, most recently from a9df684 to 6baf069 Compare October 1, 2020 11:55
test_name "Facter::Core::Execution doesn't kill process with long stderr message" do
tag 'risk:high'

file_content = <<-EOM
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can make use of ruby here, to generate the needed text, eg:

length = 1000
file_content = <<-EOM
   #!/bin/sh
   long_output=#{"Lorem ipsum dolor sit amet" * length }
   echo 'newfact=value_of_fact'
   1>&2 echo $long_output
   exit 1
EOM

And play with the length to achieve the expected output.

Also i think would be useful to add some description to this scenario and what is this test doing

Comment on lines +103 to +104
stdout_messages = +''
stderr_messages = +''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use a simple initialize?

stdout_messages = ''
stderr_messages = ''

Copy link
Contributor Author

@oanatmaria oanatmaria Oct 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they will be treated as frozen strings and this will generate an error later when we try to append the output of the command to these strings

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've started using String.new to make it clearer that I'm creating a string that isn't frozen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately i get the following error when i try to use String.new : lib/facter/custom_facts/core/execution/base.rb:104:33: C: Performance/UnfreezeString: Use unary plus to get an unfrozen string literal.

@puppetcla
Copy link

CLA signed by all contributors.

@oanatmaria oanatmaria force-pushed the long_stderr_during_execution branch 5 times, most recently from 5df2c97 to 10b7529 Compare October 2, 2020 13:38
@oanatmaria
Copy link
Contributor Author

Jenkins please test this on all

output << stdout.read
err << stderr.read
stdout_messages << out_reader.value
stderr_messages << err_reader.value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If stdout.read (on line 105) raises then ruby will raise that exception when out_reader.value is called. So it's possible for line 109 to raise something like Errno::EIO, in which case we never call err_reader.value. As a result we may never join that thread and fully read stderr. I don't know if that's going to be an actual problem though...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If out_reader.value raises an error, the error message will be logged (if execution is called with on_fail => :raise line 122). If the stdout will raise an error, the resolvers can't get the information needed for facts to be solved, the stderr message can only be used for debug purposes in this case.

end
rescue Timeout::Error
@log.debug("Timeout encounter after #{time_limit}s, killing process with pid: #{pid}")
Process.kill('KILL', pid)
out_reader.kill
err_reader.kill
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can Process.kill raise such as if the child process no longer exists? If so then kill is never called on the readers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wrapped the threads kill in an ensure block

@oanatmaria oanatmaria force-pushed the long_stderr_during_execution branch 13 times, most recently from 57f349e to 68b5d16 Compare October 6, 2020 13:40
@oanatmaria oanatmaria force-pushed the long_stderr_during_execution branch from 68b5d16 to 1a93169 Compare October 6, 2020 14:02
@mihaibuzgau mihaibuzgau merged commit e1f058e into puppetlabs:main Oct 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants