Extensions cleanup

[Frzk]

Following the comments made by @Soulou in #286, I suggest that we get rid of some useless code related to extensions.

Here is a summary that I completed with what I found in both support/ext-internal and lib (!?)

Extension	Source	Where	Decision	Comment
apcu	PECL	support/ext	✔️ KEEP (bundled)
mongodb	PECL	support/ext	✔️ KEEP (bundled)
redis	PECL	support/ext	✔️ KEEP (bundled)
calendar	PHP	support/ext-internal	✔️ KEEP (prebuilt)
ftp	PHP	support/ext-internal	✔️ KEEP (prebuilt)
gettext	PHP	support/ext-internal	✔️ KEEP (prebuilt)
gmp	PHP	support/ext-internal	✔️ KEEP (prebuilt)
imap	PHP	support/ext-internal	✔️ KEEP (prebuilt)
sodium	PHP	support/ext-internal	✔️ KEEP (prebuilt)
tidy	PHP	support/ext-internal	✔️ KEEP (prebuilt)
blackfire	External	support/ext	✔️ KEEP (prebuilt)
datadog	External	lib	✔️ KEEP (prebuilt)
newrelic	External	lib	✔️ KEEP (prebuilt)
oci8	PECL	lib	❌ DROP	Handle reqs in buildpack.
memcached	PECL	support/ext	❌ DROP	Handle reqs in buildpack.
apfd	PECL	support/ext	❌ DROP
ds	PECL	support/ext	❌ DROP
imagick	PECL	support/ext	❌ DROP
mcrypt	PECL	support/ext	❌ DROP
amqp	PECL	support/ext	❌ DROP	Handle reqs in buildpack.
event	PECL	support/ext	❌ DROP
igbinary	PECL	support/ext	❌ DROP
lua	PECL	support/ext	❌ DROP	Req. lua
scoutapm	PECL/Zend	lib	❌ DROP	Handle specs in buildpack.

• ✔️ KEEP (bundled): extensions that we want to build, ship and enable by default.
• ✔️ KEEP (prebuilt): extension that we want to build and ship, but that we don't want to enable by default.
• ❌ DROP: extensions that we don't want to care about anymore. They can be built via the php-buildpack (or multi-buildpack + apt-buildpack, as suggested in our public documentation).
• ❓ TBD: Is there any good reason to mark them as KEEP (prebuilt) instead of DROP?
  • amqp: Seems to work well with a multibuildpack (using the APT buildpack to install librabbitmq-dev). I think we can DROP.
  • event: Seems to work well out of the box. I think we can DROP.
  • igbinary: Seems to work well out of the box. I think we can DROP.
  • scoutapm: Does not work out of the box because it's a Zend extension, which means it must be loaded via zend_extension=scoutapm.so. Supporting Zend extensions means we must have specific code to handle them. It shouldn't be hard to maintain, so I think we can mark it as KEEP (prebuilt).
  • lua: Can't make it work for now (configure: error: Please reinstall the lua distribution - lua.h should be in <lua-dir>/include/). It seems we stopped supporting it a while ago (4396536).

---

• External extensions have dependencies or requirements that are hard to factorize, hence marking them all as KEEP (prebuilt).
• Some External extensions (namely datadog and newrelic are enabled and configured by using a .extra.paas.<extension_name> key in composer.json. Should this be expanded to other extensions as well? Should we keep this?
• PHP extensions (Source is set to PHP in the table above) are included in PHP. They need to be enabled with a specific configure flag. This shouldn't be hard to support, hence marking them all as KEEP (prebuilt).
• Some PHP extensions (gmp, sodium, tidy,...) still have dependencies. It's unclear to me whether we should keep downloading + compiling + packaging these dependencies or if we could apt install them.
• Some extensions marked as DROP (or that we don't support at all) have known dependencies and configure options (for example, the yaml extension requires libyaml-dev and needs to be compiled with the --with-libyaml-dir= flag):
  • Do we want to publicly document this somewhere?
  • Do we want to add code to handle this somewhere?

---

I consequently suggest the following plan:

### Plan
- [ ] 0. Decide what we do with the extensions marked as **TBD**.
- [ ] 1. Remove the `support/ext-deprecated` directory and its content.
- [ ] 2. Remove all code related to the extensions marked as **DROP** (except if we want to keep minimal support for things that we know - see question above).
- [ ] 3. Put code related to **building and packaging** the extensions marked as **KEEP (bundled)** in a directory called `support/bundled`. Each file should be named `package_<extension_name>.sh`.
- [ ] 4. Put code related to **building and packaging** the extensions marked as **KEEP (prebuilt)** in a directory called `support/prebuilt`. Each file should be named `package_<extension_name>.sh`.
- [ ] 5. Move some code used to package PHP extensions dependencies into `support/prebuilt/package_<dependency_name>.sh` scripts. Document why we need this.
- [ ] 6. Remove code related to **enabling** an extension in favor of a generic, dedicated function (to be written, the process is always the same: download the package (if necessary), copy the `.so` file, enable extension by creating `/app/vendor/php/etc/conf.d/${extension_name}.ini`).
- [ ] 7. Remove the `support/ext` and `support/ext-internal` directories after their content has been dispatched between the `support/{bundled,prebuilt}` directories or replaced by the function.
- [ ] 8. Update our public documentation to list the bundled and prebuilt extensions.
- [ ] 9. Update out internal documentation with procedures describing how to update everything (PHP, bundled and prebuilt extensions, dependencies, ...).
- [ ] 10. Figure out what to do with code related to `Scout APM`, `Datadog`, `New Relic` and `oci8`... that crawls here and there.
- [ ] 11. Figure out how to organize the `support/get_*` scripts.

@EtienneM, @Soulou, (others are welcome too) could you please let me know:

• If we agree on the table above ;
• What you think about the extensions marked as TBD ;
• What you think about the PHP extensions that have dependencies: should we keep it as they are, should we try to use apt, should we include the deps in our stacks? ;
• What you think about keeping some sort of minimal support for extensions that we don't want to build/package/ship ;
• What you think about the Plan.

Thanks a billion.
🙇‍♂️ 🙏

[EtienneM]

Thanks for this great work! 👏

What you think about the PHP extensions that have dependencies: should we keep it as they are, should we try to use apt, should we include the deps in our stacks? ;

Yes I would keep them as-is. The problem of the workflow with multibuildpacks and the apt buildpack is that it's a bit complicated for newcomers. Hence it's a bit of an entrance barrier for newcomers.

What you think about keeping some sort of minimal support for extensions that we don't want to build/package/ship ;

Even though I agree with the idea, I am kind of reluctant to drop some extensions as it would break an existing workflow for some customers. Would it be possible to gather the usage of some of these extensions to know if they are actually used or if we can drop the support with no impact?

What you think about the extensions marked as TBD ;

For those working well out of the box, I agree to simply drop them.

For those requiring the multic-buildpacks, cf. my points raised above: multibuildpacks is an entrance barrier + it's breaking currently working workflow.

Yes lua extension is not supported in PHP 8.1 but PHP 8.0 is still supported for a couple of months (https://www.php.net/supported-versions.php) so I think we should keep it until we drop support for PHP 8.0.

[leo-scalingo]

First, thanks for this great summary.

External extensions have dependencies or requirements that are hard to factorize, hence marking them all as KEEP (prebuilt).

Makes completely sense

Some External extensions (namely datadog and newrelic are enabled and configured by using a .extra.paas.<extension_name> key in composer.json. Should this be expanded to other extensions as well? Should we keep this?

They are done like this since they require some specific configuration right? Like an API key + configuration right? .extra.paas is something custom for us, if there is a way to be more generic, it should be the way we should take, but... we need to keep retrocompatibility. So what I could say is that I would not add other possibilities in the .extra.paas JSON key

PHP extensions (Source is set to PHP in the table above) are included in PHP. They need to be enabled with a specific configure flag. This shouldn't be hard to support, hence marking them all as KEEP (prebuilt).

Agreed on this.

Some PHP extensions (gmp, sodium, tidy,...) still have dependencies. It's unclear to me whether we should keep downloading + compiling + packaging these dependencies or if we could apt install them.

Could we apt install them? Can you give more details about what you would do here?

Some extensions marked as DROP (or that we don't support at all) have known dependencies and configure options (for example, the yaml extension requires libyaml-dev and needs to be compiled with the --with-libyaml-dir= flag):

We should only drop them if there is a way to install it with the standard PECL method, otherwise we may want to keep them.

Do we want to publicly document this somewhere?

If you mean how to install some specific PECL extensions, saying APT buildpack should be used with package X and that Y environment variable should be set?

Do we want to add code to handle this somewhere?

What would you think of?

[Frzk]

Some External extensions (namely datadog and newrelic are enabled and configured by using a .extra.paas.<extension_name> key in composer.json. Should this be expanded to other extensions as well? Should we keep this?

They are done like this since they require some specific configuration right? Like an API key + configuration right? .extra.paas is something custom for us, if there is a way to be more generic, it should be the way we should take, but... we need to keep retrocompatibility. So what I could say is that I would not add other possibilities in the .extra.paas JSON key

• Enabling Datadog, Scout APM and New Relic is done via the extra.paas key. Note that they can be enabled without any API key or the extra requirements (tested this afternoon).

• Enabling Blackfire requires some environment variables to be created. There's no dedicated extra.paas key. This is the first consistency issue I can see. Also there's currently absolutely nothing about Blackfire in our doc 😞

• Enabling all other extensions is done via the require key with ext-<extension_name>.

My suggestion is to make it possible to enable all extensions (including datadog, newrelic, scoutapm, and blackfire) with the require key, so we have one way to enable them all.
We could keep the extra.paas keys for a while. Maybe deprecate it with a message?

---

Some PHP extensions (gmp, sodium, tidy,...) still have dependencies. It's unclear to me whether we should keep downloading + compiling + packaging these dependencies or if we could apt install them.

Could we apt install them? Can you give more details about what you would do here?

I think we can keep compiling and packaging them, since they are PHP built-ins.
Maybe I should just review/update the packaging scripts to use apt to install the dependencies instead of downloading them from our ObjectStorage?

---

Some extensions marked as DROP (or that we don't support at all) have known dependencies and configure options (for example, the yaml extension requires libyaml-dev and needs to be compiled with the --with-libyaml-dir= flag):

We should only drop them if there is a way to install it with the standard PECL method, otherwise we may want to keep them.

Do we want to publicly document this somewhere?

If you mean how to install some specific PECL extensions, saying APT buildpack should be used with package X and that Y environment variable should be set?

Yes, that's the first idea.

Do we want to add code to handle this somewhere?

What would you think of?

I'd probably add some code to handle these known extensions and their specifics, instead of asking the customer to do the hard work:

For PECL extensions that have dependencies:

• Leverage the APT buildpack to install the dependencies,
• Copy the required libraries to the appropriate location,
• Compile the extension with the appropriate flags,
• Copy the .so file to the appropriate location, and also in CACHE_DIR for future use,
• Remove the .apt directory.

In addition, that would allow users to specify the version they want, which is a request we sometimes have.

I've started to work on this a while ago, in #341, here: https://github.com/Scalingo/php-buildpack/pull/341/files#diff-10b57d01220e227280f89077b8550047856cd636787f9dd31d8dc6773622af2cR158

This should probably be continued (and tested) 😅

It shouldn't be that hard to manage, we already have the info in the different scripts used to compile and package the extensions.

IMHO, the main question is whether it's OK to call the APT buildpack to achieve this or not?

---

Extra question: some PECL extensions are shipped and enabled by default (APCu, mongodb, redis). Do we want to continue shipping and enabling them by default? If not, I'd suggest to mark them as DROP, I'm pretty sure they'll compile just fine out of the box.

---

I've updated the table with up-to-date info.

Thanks for your interest ❤️