certain exponents cause generateKeyPair to hang

[calvinmetcalf]

• Version: 11.13.0 and also tested 10.15.0
• Platform: Darwin calvin 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64
• Subsystem: crypto

certain exponents cause crypto.generateKeyPair to hang

try running

const { publicKey, privateKey } = crypto.generateKeyPairSync('rsa', {
  modulusLength: 1024,
  publicExponent: 3,
  publicKeyEncoding: {
    type: 'spki',
    format: 'pem'
  },
  privateKeyEncoding: {
    type: 'pkcs8',
    format: 'pem'
  }
});

you'd expect it to eventually do something at least error, but it just hangs as do exponents 5 and 17, in fact the only one that seems to work is 0x10001.

This isn't a speed thing since I can generate a key like this in browser crypto and in pure javascript pretty easily.

looking at the tests it looks like 0x10001 is the only key being tested against

[sam-github]

It segfaults for me.

node/src/node_crypto.cc

Lines 6066 to 6073 in 2f1ed5c

	// 0x10001 is the default RSA exponent.
	if (exponent_ != 0x10001) {
	BignumPointer bn(BN_new());
	CHECK_NOT_NULL(bn.get());
	CHECK(BN_set_word(bn.get(), exponent_));
	if (EVP_PKEY_CTX_set_rsa_keygen_pubexp(ctx.get(), bn.get()) <= 0)
	return false;
	}

has code specific to non-default exponents, which explains why this specific path can fault but not the default exponent.

The exponent seems to be changing to a 0x555558b35110 somewhere along the way, though it is correct at BN_set_word() in the above code. Still looking, in between doing other things.