Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[debian9] libssl1.0.0 and libssl1.0.2 can end up loaded in same process causing crashes #23796

Closed
DHowett-MSFT opened this issue Oct 10, 2017 · 12 comments

Comments

@DHowett-MSFT
Copy link

On Debian 9, curl depends on libssl1.0.2. When the libssl1.0.0 and libssl1.0.2 packages are both installed on a machine, System.Security.Cryptography.Native.Openssl will preferentially load libssl.so.1.0.0. System.Net.Http.Native, however, loads libcurl.so, which in turn loads libssl.so.1.0.2.

Repro

Some combination (as yet undetermined; the repro is "open PowerShell 6.0.0-beta.8, wait a while") of interacting with the members of these assemblies, or of passing SSL contexts between System.Net.Http and System.Security.Cryptography.Openssl will trigger a segmentation violation and subsequent termination.

Dependency Graph

* System.Net.Http.Native
  * libcurl.so.4
    * libssl.so.1.0.2
* System.Security.Cryptography.Native.Openssl
  * libssl.so.1.0.0

Resolution

Not sure.

  • Probe for libssl symbols, fall back to dlopen if none are found?
  • Prefer libssl.so.1.0.2?
@DHowett-MSFT
Copy link
Author

Ah: the callstack shows a callback traversing up from ssl 1.0.2, through libcoreclr, back down to ssl 1.0.0:

  * frame #0: 0x00007fff2e3d5ddd libcrypto.so.1.0.0`___lldb_unnamed_symbol1031$$libcrypto.so.1.0.0 + 13
    frame dotnet/corefx#1: 0x00007fff2e3cfe6c libcrypto.so.1.0.0`X509_verify_cert + 1036
    frame dotnet/corefx#2: 0x00007fff82287296
    frame dotnet/corefx#3: 0x00007fff80c9ce57
    frame dotnet/corefx#4: 0x00007fff80c9cca3
    frame dotnet/corefx#5: 0x00007fff82285606
    frame dotnet/corefx#6: 0x00007ffff634af6e libcoreclr.so`UMThunkStub + 273
    frame dotnet/corefx#7: 0x00007fff560888c6 libssl.so.1.0.2`___lldb_unnamed_symbol299$$libssl.so.1.0.2 + 310
    frame dotnet/corefx#8: 0x00007fff5606332c libssl.so.1.0.2`___lldb_unnamed_symbol22$$libssl.so.1.0.2 + 716
    frame dotnet/corefx#9: 0x00007fff560682ea libssl.so.1.0.2`___lldb_unnamed_symbol34$$libssl.so.1.0.2 + 2778

@karelz
Copy link
Member

karelz commented Oct 11, 2017

@bartonjs @janvorli @Priya91 @wfurt any thoughts here?

@janvorli
Copy link
Member

I think I'll need to add a way to enable overriding the libssl we load (via an env var). I cannot see a reasonable automatic way to resolve the problem. If I switched the search order (first 1.0.2, then 1.0.0), then it would create the same potential issue on other distros where curl is built against libssl 1.0.0, but someone installs the 1.0.2 side by side with it.

@DHowett-MSFT
Copy link
Author

Through some combination of dlsym(RTLD_NEXT, ... and dladdr you can probe for an already-loaded libssl, get its path and perhaps verify its version. From there, S.S.Cryptography.Native could use the existing loaded libssl to resolve further symbols.

Would that be an appropriate solution?

@janvorli
Copy link
Member

@DHowett-MSFT the issue is that the libcurl doesn't have to be loaded yet at the point when we load libssl. The libcurl is loaded only if System.Net.Http.dll is used and libssl when System.Security.Cryptography.dll is used.

@janvorli
Copy link
Member

Maybe we could try to change the System.Security.Cryptography.Native.OpenSSL.so to load libssl indirectly by loading libcurl, but that's kind of ugly since it pulls quite a lot of other libraries into the process and if someone is using just the crypto stuff and no https, then it would be waste of space.

@wfurt
Copy link
Member

wfurt commented Oct 11, 2017

those pages would be easily releasable as OS knows it is backed by file.
Since RTLD_LOCAL should be default, there really should not be conflict with symbols, right?
So even if loaded twice, we would only get crash if for example X509 structure is not binary compatible. Would it be reasonable to perform some check when either ssl or curl is loaded?
I think throwing meaningful error would be better than mysterious crash.

@DHowett-MSFT
Copy link
Author

DHowett-MSFT commented Oct 11, 2017

In this case, it looks like that’s what’s happening. A callback through 1.0.2 is ending up serviced by a 1.0.0 consumer with 1.0.0 X509 APIs. RTLD_LOCAL won’t really help with that.

@janvorli
Copy link
Member

One important thing to note is that having these two libraries is a non-standard situation. There is no official 1.0.0 package for Debian 9, only 1.0.2. That's why I was suggesting treating it by adding a way to explicitly override the selection in the unlikely case someone has both versions due to some unknown reason. Please correct me if this assumption is wrong and it is a more wide-spread case.
@DHowett-MSFT do you have any idea how the 1.0.0 package got into your Debian 9?

@DHowett-MSFT
Copy link
Author

It looks like it came in when I transitioned from Debian 8. I have a few packages currently installed that depend on it, many of which seem important.

Thanks for the tip on it being non-standard, however. I could have sworn it was standard-issue!

Since that's the case, perhaps there's little value in fixing this issue.

@Ghostbird
Copy link

@DHowett-MSFT I had the same problem with the transition from Debian 8. The weirdest thing was that there was no error on the .NET side. My API terminated as if performing an expected shutdown (exit code 0). That is a pretty major bug in my books, but the underlying cause was my misconfigured system. Uninstalling libssl1.0.0 was the solution for me.

@bartonjs
Copy link
Member

Since we've already marked dotnet/corefx#24891 for inclusion in the 2.1 release, closing this copy of the issue.

@msftgits msftgits transferred this issue from dotnet/corefx Jan 31, 2020
@msftgits msftgits added this to the 2.1.0 milestone Jan 31, 2020
@msftbot msftbot bot locked as resolved and limited conversation to collaborators Dec 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants