Backport of ephemerals: allow root ephemeral outputs in stacks and test into v1.14 #37820
+5,378
−2,312
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Prepare main branch for post-v1.14 development * Update pull_request_template.md to reflect how main == v1.15 now
* Add test coverage for `parseReattachProviders` * Add unhappy path test cases to parseReattachProviders test * Add `isProviderReattached` function, to help identify when a reattached provider is being used for PSS * Move reattach config-related code into its own package * Make calling code not need to know what the ENV is for reattach providers * Refactor IsProviderReattached to use pre-existing logic * Add test case, make spell check happy with US English * Add headers * Make calling code responsible for reading in the related ENV * Add godoc comment to TF_REATTACH_PROVIDERS const * Update internal/getproviders/reattach/reattach.go
…ired attrs (#37626) * actions: improve handling of null config so we can properly report missing required arguments There are various methods in terraform that will let you know if you are missing required attributes - none of which work with nil inputs. This commit changes the handling, passing in an empty object if the config was null, and adding additional context to the error message when the config block is missing.
…37653) * actions: allow invoke only on local execution while TFC adds support * actually return early
Handle `-help` usage in stacks plugin
docs(destroying): fix typos
We don't need to plan any actions during query plan mode. Test stolen from DanielMSchmidt, thank you!
…als and resources (#37663) * evaluate: return diagnostics instead of unknown for uninitialised locals and resources * changelog * also input variables
…r accessing a state store. (#37665) * Add method to allow accessing factories from locks that are in memory * Create new getStateStoreProviderFactory method for accessing a factory from config * Update getStateStoreProviderFactory to use in memory locks instead of reading them from the deps lock file. Update calling code to accommodate this. * Add tests for getStateStoreProviderFactory, improve errors returned from method * Update test following schema change in simple providers * Move test case into e2etest package, so we protect against environments where building binaries isn't possible * Fix issues with running test in e2etest package * Update code comments Co-authored-by: Radek Simko <[email protected]> --------- Co-authored-by: Radek Simko <[email protected]>
* Implement ReadStateBytes + WriteStateBytes * [WIP] - Testing ReadStateBytes and WriteStateBytes (#37464) * Fix nil pointer error * Add WIP test for ReadStateBytes * Move test mock to separate testing file * Update mock to send unexpected EOF when there's a problem returning data and it's not a true EOF * Add test case for when length != expected length * Add test for when trying to read state from a store type that doesn't exist * Change symbol names to lowercase * Add ability to force a diagnostic to be returned from `mockReadStateBytesClient`'s `Recv` method * Add test showing error diagnostics raised by the ReadStateBytes client are returned * Add missing header * Simplify mock by using an embedded type * Rename `mockOpts` to `mockReadStateBytesOpts` * Update existing tests to assert what arguments are passed to the RPC method call * Add mock WriteStateBytesClient which uses `go.uber.org/mock/gomock` to enable assertions about calls to Send * Add a test for WriteStateBytes that makes assertions about calls to the Send method * Update test case to explicitly test writing data smaller than the chunk size * Implement chunking in WriteStateBytes, add test case to assert expected chunking behaviour * Add generated mock for Provider_WriteStateBytesClient in protocol v6 * Update tests to use new `MockProvider_WriteStateBytesClient`, remove handwritten mock * Update code comments in test * Add tests for diagnostics and errors returned during WriteStateBytes * Add generated mock for Provider_ReadStateBytesClient in protocol v6, replace old mock * Add test case for grpc errors in ReadStateBytes, fix how error is returned * Typo in comment * Add missing warning test, rename some test cases * Update proto file definition of Read/WriteStateBytes RPCs (#37529) * Update Read/WriteStateBytes RPCs to match hashicorp/terraform-plugin-go#531 * Run `make protobuf` * Run `make generate` * Update use of `proto.ReadStateBytes_ResponseChunk` in tests * Fix how diagnostics are handled alongside EOF error, update ReadStateBytes test * More fixes - test setup was incorrect I think? I assume that a response would be returned full of zero-values when EOF is encountered. * WIP - avoid crash if chunk is nil * Sarah's updates to radek/pss-read-write (#37642) * Update code to not expect a chunk when EOF encountered * Return early if any grpc errors are encountered during ReadStateBytes * Close the stream with CloseSend once everything's read without error. Add test case about handling grpc errors from CloseSend. * Fix test case about warnings: We would expect to receive a chunk with data alongside the warning and have a normal closing of the stream after EOF * Add log line, remove unneeded type info * Implement configurable state chunk size * handle metadata in WriteStateBytes correctly * validate chunk sizes received from provider * ReadStateBytes: avoid early return on warnings --------- Co-authored-by: Sarah French <[email protected]>
* Fix lack of protoc-gen-go-grpc This appears to have been removed in https://github.com/hashicorp/terraform/pull/29492/files#diff-0329966ed7ce7286cfaf91161ac003af419cc566dd6420bb07670f9e4a389c81 accidentally following linting? * Fix Args to include all arguments The docs say "Args holds command line arguments, including the command as Args[0]." * Add TODO for swapping to non-deprecated repo for protoc-gen-go
…of `github.com/golang/protobuf/protoc-gen-go` (#37685) * Reformat Args in steps definitions * Swap to using the new version of protoc-gen-go that doesn't include grpc support (instead it relies on protoc-gen-go-grpc). Update command flags to reflect this change in tooling. See this comment for explanation: golang/protobuf#1070 (comment) * Run `make protobuf` with problem generate steps commented out. * Use `require_unimplemented_servers=false` to minimise changes at the time of upgrading tooling In future we can navigate the consequences of this in its own PR. * WIP * Fix out/opt typo, add remaining flags once terraform1.proto was being found ok * Run `make protobuf` * `go get google.golang.org/grpc/cmd/[email protected]` && `go mod tidy` This version includes a feature that allows the copyright comment in the .proto file to be pulled across to generated _grpc.pb.go files. * Run `make protobuf` with new version of protoc-gen-go-grpc that allows copyright comment to be copied to generated output files * Potential fix for `internal/rpcapi/dynrpcserver/generator`
During the latest refactor, the wrong schema was fetched when generating configuration from a list resource. Having the list resource schema instead of the managed resource schema prevented the config generation from matching the computed attributes within the state, causing them to show up in the configuration. It turns out the tests for this didn't have computed attributes, so the accidental swap during refactoring didn't cause the tests to fail. A small addition to the tests adding an `id` attribute picks the regression.
turns out the invalid id attributes were captured in the command test, so removing them caused that to fail too.
get correct schema when generating query config
AWS European Sovereign Cloud support, bump aws-sdk-go-base
* actions: connect invoke flag with TFC * use latest go-tfe
… code from .proto files (#37737) * Remove `require_unimplemented_servers=false` option when generating code from proto files. Run `make protobuf`. This allows require_unimplemented_servers to default to true, and the generated code shows the impacts of that. * Update generator script to embed 'UnimplementedFoobarServer' structs and skip generating code for the `mustEmbedUnimplementedFoobarServer` method that is implemented via that embedding. * Run `make generate` * Embed UnimplementedProviderServer into implementations of ProviderServer * Embed UnimplementedProvisionerServer into implementations of ProvisionerServer
…37751) * Fix crash when showing a cloud plan without a cloud backend * Add changelog
Bump go-getter v1.8.2
* Implement Lock and Unlock methods on grpcClient * Add tests for Lock and Unlock methods * Reuse same lock info variable
This PR replaces the old Slack channel id in our release tooling configuration with the new one.
…cs (#37762) * Replace `errBackendLocalRead` constant with custom error of same name * Replace `errBackendNoExistingWorkspaces` constant with custom error of same name * Replace `errBackendMigrateLocalDelete` constant with custom error of same name * Replace `errBackendNewUnknown` constant with custom error of same name * Replace `errBackendSavedUnknown` constant with custom error of same name * Replace `errBackendClearSaved` constant with custom error of same name * Replace `errBackendInit` constant with custom error of same name * Replace `errBackendInitCloud` constant with custom error of same name * Replace `errBackendWriteSaved` constant with custom error of same name * Move `migrateOrReconfigDiag` next to custom errors * Replace `errBackendInit` custom error with `func errBackendInitDiag` for creating diagnostics directly * Replace `errBackendInitCloud` custom error with `func errBackendInitCloudDiag` for creating diagnostics directly * Replace `errBackendWriteSaved` custom error with `func errBackendWriteSavedDiag` for creating diagnostics directly * Add new usage of errBackendWriteSavedDiag; this is another place where failing to write to backend state files happens * Add godoc comments * Ensure errors aren't repeated in terminal output, and ensure warnings are not lost.
…37705) Bumps the github-actions-breaking group with 2 updates: [actions/setup-go](https://github.com/actions/setup-go) and [actions/github-script](https://github.com/actions/github-script). Updates `actions/setup-go` from 5.5.0 to 6.0.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@d35c59a...4469467) Updates `actions/github-script` from 7.0.1 to 8.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@60a0d83...ed59741) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-breaking - dependency-name: actions/github-script dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-breaking ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… updates (#37706) Bumps the github-actions-backward-compatible group with 2 updates: [actions/cache](https://github.com/actions/cache) and [miniscruff/changie-action](https://github.com/miniscruff/changie-action). Updates `actions/cache` from 4.2.4 to 4.3.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@0400d5f...0057852) Updates `miniscruff/changie-action` from 2.0.0 to 2.1.0 - [Release notes](https://github.com/miniscruff/changie-action/releases) - [Changelog](https://github.com/miniscruff/changie-action/blob/main/CHANGELOG.md) - [Commits](miniscruff/changie-action@6dcc253...5036dff) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions-backward-compatible - dependency-name: miniscruff/changie-action dependency-version: 2.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions-backward-compatible ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Sebastien Dionne <[email protected]>
…econfigure` flag) while using `state_store` (#37732) * Minor fixes in diagnostics This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places. * Rename test to make it specific to use of backend block in config * Update initBackend to accept whole initArgs collection * Only process --backend-config data, when setting up a `backend`, if that data isn't empty * Simplify how mock provider factories are made in tests * Update mock provider's default logic to track and manage existing workspaces * Add `ProviderSchema` method to `Pluggable` structs. This allows calling code to access the provider schema when using provider configuration data. * Add function for converting a providerreqs.Version to a hashicorp/go-version Version. This is needed for using locks when creating the backend state file. * Implement initial version of init new working directories using `stateStore_C_s`. Default to creating the default workspace if no workspaces exist. * Update test fixtures to match the hashicorp/test mock provider used in PSS tests * Allow tests to obtain locks that include `testingOverrides` providers. The `testingOverrides` field will only be set in tests, so this should not impact end users. * Add tests showing TF can initialize a working directory for the first time (and do the same when forced by -reconfigure flag). Remove replaced tests. * Add -create-default-workspace flag, to be used to disable creating the default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests. * Allow reattached providers to be used during init for PSS * Rename variable to `backendHash` so relation to `backend` is clearer * Allow `(m *Meta) Backend` to return warning diagnostics * Protect against nil testingOverrides in providerFactoriesFromLocks * Add test case seeing what happens if default workspace selected, doesn't exist, but other workspaces do exist. The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`. * Address code consistency check failure on PR * Refactor use of mock in test that's experiencing EOF error... * Remove test that requires test to supply input for user prompt This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input. * Allow -create-default-workspace to be used regardless of whether input is enabled or disabled * Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable * Responses to feedback, including making testStdinPipe helper log details of errors copying data to stdin. Note: We cannot call t.Fatal from a non-test goroutine. * Use Errorf instead * Allow backend state files to not include version data when a builtin or reattached provider is in use. * Add clarifying comment about re-attached providers when finding the matching entry in required_providers * Report that the default workspace was created to the view * Refactor: use error comparison via `errors.Is` to identify when no workspaces exist. * Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseInit func. * Validate that PSS-related flags can only be used when experiments are enabled, enforce coupling of PSS-related flags when in use. * Slight rewording of output message about default workspace * Update test to assert new output about default workspace
…nfiguration. (#37777) * Pull determining of PSS provider's version from current locks into a separate method * Add code for identifying when config and provider version match existing backend state (i.e. no changes) * Update test - locks are now needed before it hits expected error diag return * Add test showing successful init when no config changes are detected. * Update `getStateStorageProviderVersion` to return nil versions for builtin and re-attached providers. This makes comparison easier when determining if config has changed since last init. * Add test coverage for `getStateStorageProviderVersion` * Move testing fixtures around, preparing for different types of changed state_store config changes being tested * Add test showing that changing the state_store config is detected as a change, but handling this scenario isn't implemented yet * Update hashes in test fixture backend state file to be accurate Previously dummy values were fine, but as tests using hashes to identify changes these values need to be accurate! * Update existing test cases so that Terraform uses the same test provider version as described in the backend state file fixture for the test. * Add test showing that changing the PSS provider's config is detected as a change, but handling this scenario isn't implemented yet * Add test showing that swapping to a different state storage implementation in the same provider is detected as a change, but handling this scenario isn't implemented yet * Add test showing that changing the provider used for PSS is detected as a change, but handling this scenario isn't implemented yet * Add test showing that upgrading a provider is detected as a change, but handling this scenario isn't implemented yet * Update test to use v1.2.3 for consistency with other tests Just to avoid any confusion if copy-pasting happens in future. * More corrections to existing test fixtures - unset config should be null, and replace dummy hash values with correct values. * Fix test for using -reconfigure with state_store; the default workspace would already exist in this scenario * Update TestInit_stateStore_configUnchanged to assert that init was a no-op for backend state * Remove unused fixture * Remove test that's replaced by new tests in command/init_test.go * Replace old references to deleted "state-store-changed" test fixture & update test to not expect a value for region attr in provider config * Make test fixture coupling a little more understandable * Refactor detection of no need to migrate into a function * Add TODO about more involved provider version change tests We will allow downgrades to succeed as long as the schema version number is unchanged * Update (configs.StateStore)Hash method to return a single hash that's impacted by: state store config, provider config, state store type, provider source * Update calling code and test helper code to reflect that the nested provider block no longer has its own hash * Remove test; there is now a single hash that SHOULD be affected by the provider block! * Also use provider name, from config, in hash * Update tests to reflect changes in how hashes are made * Remove unused `stateStoreConfigNeedsMigration` function * Remove duplicate isProviderReattached function. * Fixes to affected tests * Allow provider version to impact the state storage hash, update impacted tests and test fixtures * Update tests that now require locks data to be present in test setup * Update comment for accuracy * Fixes to other test fixtures - remove excess hash field, set hash to 0 to indicate they're not set accurately. * Make upgrade test actually use upgrade code path * Add lock files to test fixture directories that represent a project that's had a successful prior init using PSS
…plugin-go (#37647) * Update protoc version in downloader script * go get google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.5.1 This matched terraform-plugin-go * make protobuf * Run `make protobuf` * Update generator to handle generic types from google.golang.org/grpc Looks like this was added in v1.69.3 in grpc/grpc-go#7057 ? * Run `make generate` * Fix "cannot infer Res" compile error - more usage of generics * More fixing compile errors due to switching to use of a generic * Make putting `google.golang.org/grpc` import into generated files conditional * Run `make generate` * Update more places where generics now need to be used * Update generator to handle any types from google.golang.org/grpc in same switch case.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This PR is auto-generated from #37813 to be assessed for backporting due to the inclusion of the label 1.14-backport.
The below text is copied from the body of the original PR.
Backport after 1.14.0 release.
We currently prevent root modules from exposing ephemeral values via outputs. This makes total sense as the ephemeral values shouldn't be saved or exposed anywhere.
However, in both stacks and terraform test, we sometimes pretend that a module is a root module when it isn't really. This PR disables the ephemeral check within Terraform Test and Terraform Stacks. For Stacks this is fine, as there are additional checks at the higher level to prevent stack outputs themselves being ephemeral. For Terraform Test we want people to actually test non-root modules, so we do need Core to support ephemeral outputs at the root module as the module being tested might not be intended to be consumed as a root module anyway.
Fixes #37805
Target Release
1.15.x
Rollback Plan
Changes to Security Controls
Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.
CHANGELOG entry
Overview of commits
47b0382
b6a6407