Domain+CA authority

Is it possible to pin cert to domain+CA only? E.g. Domain=example.com and CA=Let's Encrypt. This way the burden of securing a cert is shifted from example.com to Let's Encrypt and allows for easier cert rotation in the event example.com cert gets compromised