diff --git a/docs/markdown/Features-module.md b/docs/markdown/Features-module.md new file mode 100644 index 000000000000..a0b07ad4b577 --- /dev/null +++ b/docs/markdown/Features-module.md @@ -0,0 +1,259 @@ +# Features module + +## Overview + +Dealing with numerous CPU features through C and C++ compilers is a challenging task, +especially when aiming to support massive amount of CPU features for various architectures and multiple compilers +Additionally, supporting both baseline features and additional features dispatched at runtime presents another dilemma. + +Another issue that may arise is simplifying the implementations of generic interfaces while keeping the dirty work laid +on the build system rather than using nested namespaces or recursive sources, relying on pragma or compiler targets attributes +on top of complicated precompiled macros or meta templates, which can make debugging and maintenance difficult. + +While this module doesn't force you to follow a specific approach, it instead paves the way to count on a +practical multi-targets solution that can make managing CPU features easier and more reliable. + +In a nutshell, this module helps you deliver the following concept: + +```C +// Brings the headers files of enabled CPU features +#ifdef HAVE_SSE + #include +#endif +#ifdef HAVE_SSE2 + #include +#endif +#ifdef HAVE_SSE3 + #include +#endif +#ifdef HAVE_SSSE3 + #include +#endif +#ifdef HAVE_SSE41 + #include +#endif +#ifdef HAVE_POPCNT + #ifdef _MSC_VER + #include + #else + #include + #endif +#endif +#ifdef HAVE_AVX + #include +#endif + +#ifdef HAVE_NEON + #include +#endif + +// MTARGETS_CURRENT defined as compiler argument via `features.multi_targets()` +#ifdef MTARGETS_CURRENT + #define TARGET_SFX(X) X##_##MTARGETS_CURRENT +#else + // baseline or when building source without this module. + #define TARGET_SFX(X) X +#endif + +void TARGET_SFX(my_kernal)(const float *src, float *dst) +{ +#ifdef HAVE_AVX512F + // defeintions for implied features alawys present + // no matter the compiler is + #ifndef HAVE_AVX2 + #error "Alawys defined" + #endif +#elif defined(HAVE_AVX2) && defined(HAVE_FMA3) + #ifndef HAVE_AVX + #error "Alawys defined" + #endif +#elif defined(HAVE_SSE41) + #ifndef HAVE_SSSE3 + #error "Alawys defined" + #endif +#elif defined(HAVE_SSE2) + #ifndef HAVE_SSE2 + #error "Alawys defined" + #endif +#elif defined(HAVE_ASIMDHP) + #if !defined(HAVE_NEON) || !defined(HAVE_ASIMD) + #error "Alawys defined" + #endif +#elif defined(HAVE_ASIMD) + #ifndef HAVE_NEON_VFPV4 + #error "Alawys defined" + #endif +#elif defined(HAVE_NEON_F16) + #ifndef HAVE_NEON + #error "Alawys defined" + #endif +#else + // fallback to C scalar +#endif +} +``` + +From the above code we can deduce the following: +- The code is written on top features based definitions rather than counting clusters or + features groups which gives the code more readability and flexibility. + +- Avoid using compiler built-in defeintions no matters the enabled arguments allows you + to easily manage the enabled/disabled features and to deal with any kind of compiler or features. + Since compilers like MSVC for example doesn't provides defeintions for all CPU features. + +- The code is not aware of how its going to be build it, that gives the code a great prodiblity to + manage the generated objects which allow raising the baseline features at any time + or reduce and increase the additional dispatched features without changing the code. + +- Allow building a single source multiple of times simplifying the implementations + of generic interfaces. + + +## Usage + +To use this module, just do: **`features = import('features')`**. The +following functions will then be available as methods on the object +with the name `features`. You can, of course, replace the name `features` +with anything else. + +### features.new() +```meson +features.new(string, int, + implies: FeatureOject | FeatureObject[] = [], + group: string | string[] = [], + detect: string | {} | (string | {})[] = [], + args: string | {} | (string | {})[] = [], + test_code: string | File = '', + extra_tests: {string: string | file} = {}, + disable: string = '' + ) -> FeatureObject +``` + +This function plays a crucial role in the Features Module as it creates +a new `FeatureObject` instance that is essential for the functioning of +other methods within the module. + +It takes two required positional arguments. The first one is the name of the feature, +and the second is the interest level of the feature, which is used by +the sort operation and priority of conflicting arguments. +The rest of the kwargs arguments are explained as follows: + +* `implies` **FeatureOject | FeatureObject[] = []**: One or an array of features objects + representing predecessor features. + +* `group` **string | string[] = []**: Optional one or an array of extra features names + to be added as extra definitions that can passed to source. + +* `args` **string | {} | (string | {})[] = []**: Optional one or an array of compiler + arguments that are required to be enabled for this feature. + Each argument can be a string or a dictionary that holds four items that allow dealing + with the conflicts of the arguments of implied features: + - `val` **string**: string, the compiler argument. + - `match` **string | empty**: regex to match the arguments of implied features + that need to be filtered or erased. + - `mfilter` **string | empty**: regex to find certain strings from the matched arguments + to be combined with `val`. If the value of `mfilter` is empty or undefined, + any matches triggered by the value of `match` will not be combined with `val`. + - `mjoin` **string | empty**: a separator used to join all the filtered arguments. + If it's empty or undefined, the filtered arguments will be joined without a separator. + +* `detect` **string | {} | (string | {})[] = []**: Optional one or an array of features names + that required to be detect on runtime. If no features sepecfied then the values of `group` + will be used if its provides otherwise the name of the feature will be used instead. + Similar to `args`, each feature name can be a string or a dictionary that holds four items + that allow dealing with the conflicts of the of implied features names. + See `features.multi_targets()` or `features.test()` for more clearfications. + +* `test_code` **string | File = ''**: Optional C/C++ code or the path to the source + that needs to be tested against the compiler to consider this feature is supported. + +* `extra_tests` **{string: string | file} = {}**: Optional dictionary holds extra tests where + the key represents the test name, which is also added as a compiler definition if the test succeeded, + and the value is C/C++ code or the path to the source that needs to be tested against the compiler. + +* `disable` **string = ''**: Optional string to consider this feature disabled. + +Returns a new instance of `FeatureObject`. + +Example: + +```Meson +cpu = host_machine.cpu_family() +features = import('features') + +ASIMD = features.new( + 'ASIMD', 1, group: ['NEON', 'NEON_VFPV4', 'NEON_VFPV4'], + args: cpu == 'aarch64' ? '' : [ + '-mfpu=neon-fp-armv8', + '-march=armv8-a+simd' + ], + disable: cpu in ['arm', 'aarch64'] ? '' : 'Not supported by ' + cpu +) +# ARMv8.2 half-precision & vector arithm +ASIMDHP = features.new( + 'ASIMDHP', 2, implies: ASIMD, + args: { + 'val': '-march=armv8.2-a+fp16', + # search for any argument starts with `-match=` + 'match': '-march=', + # gets any string starts with `+` and apended to the value of `val` + 'mfilter': '\+.*' + } +) +## ARMv8.2 dot product +ASIMDDP = features.new( + 'ASIMDDP', 3, implies: ASIMD, + args: {'val': '-march=armv8.2-a+dotprod', 'match': '-march=.*', 'mfilter': '\+.*'} +) +## ARMv8.2 Single & half-precision Multiply +ASIMDFHM = features.new( + 'ASIMDFHM', 4, implies: ASIMDHP, + args: {'val': '-march=armv8.2-a+fp16fml', 'match': '-march=.*', 'mfilter': '\+.*'} +) +``` + +### features.test() +```meson +features.test(FeatureObject..., + anyfet: bool = false, + force_args: string | string[] | empty = empty, + compiler: Compiler | empty = empty, + cached: bool = true, + ) -> {} +``` + +Test a one or set of features against the compiler and returns a dictionary +contains all required information that needed for building a source that +requires these features. + +### features.multi_targets() +```meson +features.multi_targets(string, ( + str | File | CustomTarget | CustomTargetIndex | + GeneratedList | StructuredSources | ExtractedObjects | + BuildTarget + )..., + dispatch: (FeatureObject | FeatureObject[])[] = [], + baseline: empty | FeatureObject[] = empty, + prefix: string = '', + compiler: empty | compiler = empty, + cached: bool = True + ) [{}[], StaticLibrary[]] +``` + + +### features.sort() +```meson +features.sort(FeatureObject..., reverse: bool = false) : FeatureObject[] +``` + +### features.implicit() +```meson +features.implicit(FeatureObject...) : FeatureObject[] +``` + +### features.implicit_c() +```meson +features.implicit_c(FeatureObject...) : FeatureObject[] +``` + diff --git a/docs/sitemap.txt b/docs/sitemap.txt index b6d57a493acc..bd4c97b09e18 100644 --- a/docs/sitemap.txt +++ b/docs/sitemap.txt @@ -59,6 +59,7 @@ index.md Windows-module.md i18n-module.md Wayland-module.md + Features-module.md Java.md Vala.md D.md diff --git a/mesonbuild/modules/feature/__init__.py b/mesonbuild/modules/feature/__init__.py deleted file mode 100644 index 6433d98eb442..000000000000 --- a/mesonbuild/modules/feature/__init__.py +++ /dev/null @@ -1,41 +0,0 @@ -# Copyright (c) 2023, NumPy Developers. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are -# met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# -# * Redistributions in binary form must reproduce the above -# copyright notice, this list of conditions and the following -# disclaimer in the documentation and/or other materials provided -# with the distribution. -# -# * Neither the name of the NumPy Developers nor the names of any -# contributors may be used to endorse or promote products derived -# from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import typing as T - -from .module import Module - -if T.TYPE_CHECKING: - from ...interpreter import Interpreter - -def initialize(interpreter: 'Interpreter') -> Module: - return Module() - diff --git a/mesonbuild/modules/feature/module.py b/mesonbuild/modules/feature/module.py deleted file mode 100644 index 7b2e4a138b17..000000000000 --- a/mesonbuild/modules/feature/module.py +++ /dev/null @@ -1,481 +0,0 @@ -# Copyright (c) 2023, NumPy Developers. -# All rights reserved. - -import typing as T -import os - -from ... import mlog, build -from ...compilers import Compiler -from ...mesonlib import File, MesonException -from ...interpreter.type_checking import NoneType -from ...interpreterbase.decorators import ( - noKwargs, noPosargs, KwargInfo, typed_kwargs, typed_pos_args, - ContainerTypeInfo, permittedKwargs -) -from .. import ModuleInfo, NewExtensionModule, ModuleReturnValue -from .feature import FeatureObject, ConflictAttr -from .utils import test_code, get_compiler - -if T.TYPE_CHECKING: - from typing import TypedDict - from ...interpreterbase import TYPE_var, TYPE_kwargs - from .. import ModuleState - from .feature import FeatureKwArgs - - class TestKwArgs(TypedDict): - compiler: T.Optional[Compiler] - force_args: T.Optional[T.List[str]] - any: T.Optional[bool] - -class Module(NewExtensionModule): - INFO = ModuleInfo('feature', '0.1.0') - - def __init__(self) -> None: - super().__init__() - self.methods.update({ - 'new': self.new_method, - 'test': self.test_method, - 'implicit': self.implicit_method, - 'implicit_c': self.implicit_c_method, - 'sort': self.sort_method, - 'multi_target': self.multi_target_method, - }) - # TODO: How to store and load from files in meson? - self.cached_tests = {} - - def new_method(self, state: 'ModuleState', - args: T.List['TYPE_var'], - kwargs: 'TYPE_kwargs') -> FeatureObject: - return FeatureObject(state, args, kwargs) - - @typed_pos_args('feature.test', varargs=FeatureObject, min_varargs=1) - @typed_kwargs('feature.test', - KwargInfo('compiler', (NoneType, Compiler)), - KwargInfo('anyfet', bool, default = False), - KwargInfo( - 'force_args', (NoneType, str, ContainerTypeInfo(list, str)), - listify=True - ), - ) - def test_method(self, state: 'ModuleState', - args: T.Tuple[T.List[FeatureObject]], - kwargs: 'TestKwArgs' - ) -> T.List[T.Union[bool, T.Dict[str, T.Any]]]: - - features = args[0] - features_set = set(features) - anyfet = kwargs['anyfet'] - compiler = kwargs.get('compiler') - if not compiler: - compiler = get_compiler(state) - - force_args = kwargs['force_args'] - if force_args is not None: - # removes in empty strings - force_args = [a for a in force_args if a] - - cached, test_result = self.test( - state, features=features_set, - compiler=compiler, - anyfet=anyfet, - force_args=force_args - ) - if not test_result['is_supported']: - if test_result['is_disabled']: - label = mlog.yellow('disabled') - else: - label = mlog.yellow('Unsupported') - else: - label = mlog.green('Supported') - if anyfet: - unsupported = [ - fet.name for fet in sorted(features_set) - if fet.name not in test_result['features'] - ] - if unsupported: - unsupported = ' '.join(unsupported) - label = mlog.green(f'Parial support, missing({unsupported})') - - features_names = ' '.join([f.name for f in features]) - log_prefix = f'Test features "{mlog.bold(features_names)}" :' - cached_msg = f'({mlog.blue("cached")})' if cached else '' - if not test_result['is_supported']: - mlog.log(log_prefix, label, 'due to', test_result['fail_reason']) - else: - mlog.log(log_prefix, label, cached_msg) - return [test_result['is_supported'], test_result] - - def test(self, state, features: T.Set[FeatureObject], - compiler: 'Compiler', - anyfet: bool = False, - force_args: T.Optional[T.Tuple[str]] = None, - _caller: T.Set[FeatureObject] = set() - ) -> T.Tuple[bool, T.Dict[str, T.Union[str, T.List[str]]]]: - # cached hash should inveolov all implied features - # since FeatureObject is mutable object. - implied_features = self.implicit(features) - test_hash = hash(( - tuple(sorted(features)), - tuple(sorted(implied_features)), - compiler, anyfet, - (-1 if force_args is None else tuple(force_args)) - )) - result = self.cached_tests.get(test_hash) - if result is not None: - return True, result - - all_features = sorted(implied_features.union(features)) - if anyfet: - cached, test_result = self.test( - state, features=features, - compiler=compiler, - force_args=force_args - ) - if test_result['is_supported']: - self.cached_tests[test_hash] = test_result - return False, test_result - - features_any = set() - for fet in all_features: - _, test_result = self.test( - state, features={fet,}, - compiler=compiler, - force_args=force_args - ) - if test_result['is_supported']: - features_any.add(fet) - - _, test_result = self.test( - state, features=features_any, - compiler=compiler, - force_args=force_args - ) - self.cached_tests[test_hash] = test_result - return False, test_result - - # For multiple features, it important to erase any features - # implied by another to avoid duplicate testing since - # implied already tested also we use this set to genrate - # unque target name that can be used for multiple targets - # build. - prevalent_features = features.difference(implied_features) - if len(prevalent_features) == 0: - # It happens when all features imply each other. - # Set the highest interested feature - prevalent_features = sorted(features)[-1:] - else: - prevalent_features = sorted(prevalent_features) - - prevalent_names = [fet.name for fet in prevalent_features] - # prepare the result dict - test_result = { - 'target_name': '__'.join(prevalent_names), - 'prevalent_features': prevalent_names, - 'features': [fet.name for fet in all_features], - 'args': [], - 'detect': [], - 'defines': [], - 'undefines': [], - 'is_supported': True, - 'is_disabled': False, - 'fail_reason': '', - } - def fail_result(fail_reason, is_disabled = False, - result_dict = test_result.copy()): - result_dict.update({ - 'is_supported': False, - 'is_disabled': is_disabled, - 'fail_reason': fail_reason, - 'features': [] - }) - self.cached_tests[test_hash] = result_dict - return False, result_dict - - # since we allows features to imply each other - # items of `features` may part of `implied_features` - _caller = _caller.union(prevalent_features) - predecessor_features = implied_features.difference(_caller) - for fet in sorted(predecessor_features): - _, pred_result = self.test( - state, features={fet,}, - compiler=compiler, - force_args=force_args, - _caller=_caller - ) - if not pred_result['is_supported']: - reason = f'Implied feature "{fet.name}" ' - pred_disabled = pred_result['is_disabled'] - if pred_disabled: - fail_reason = reason + 'is disabled' - else: - fail_reason = reason + 'is not supported' - return fail_result(fail_reason, pred_disabled) - - for k in ['defines', 'undefines']: - values = test_result[k] - pred_values = pred_result[k] - values += [v for v in pred_values if v not in values] - - # Sort based on the lowest interest to deal with conflict attributes - # when combine all attributes togathers - conflict_attrs = ['detect'] - if force_args is None: - conflict_attrs += ['args'] - else: - test_result['args'] = force_args - - for fet in all_features: - for attr in conflict_attrs: - values: T.List[ConflictAttr] = getattr(fet, attr) - accumulate_values = test_result[attr] - for conflict in values: - conflict_val: str = conflict.val - if not conflict.match: - accumulate_values.append(conflict_val) - continue - # select the acc items based on the match - new_acc: T.List[str] = [] - for acc in accumulate_values: - # not affected by the match so we keep it - if not conflict.match.match(acc): - new_acc.append(acc) - continue - # no filter so we totaly escape it - if not conflict.mfilter: - continue - filter_val = conflict.mfilter.findall(acc) - # no filter match so we totaly escape it - if not filter_val: - continue - conflict_val += conflict.mjoin.join(filter_val) - new_acc.append(conflict_val) - test_result[attr] = new_acc - - test_args = compiler.has_multi_arguments - args = test_result['args'] - if args: - supported_args, test_cached = test_args(args, state.environment) - if not supported_args: - return fail_result( - f'Arguments "{", ".join(args)}" are not supported' - ) - - for fet in prevalent_features: - if fet.disable: - return fail_result( - f'{fet.name} is disabled due to "{fet.disable}"', - fet.disable - ) - - if fet.test_code: - _, tested_code, _ = test_code( - state, compiler, args, fet.test_code - ) - if not tested_code: - return fail_result( - f'Compiler fails against the test code of "{fet.name}"' - ) - - test_result['defines'] += [fet.name] + fet.group - for extra_name, extra_test in fet.extra_tests.items(): - _, tested_code, _ = test_code( - state, compiler, args, extra_test - ) - k = 'defines' if tested_code else 'undefines' - test_result[k].append(extra_name) - - self.cached_tests[test_hash] = test_result - return False, test_result - - @permittedKwargs(build.known_stlib_kwargs | { - 'dispatch', 'baseline', 'prefix' - }) - @typed_pos_args('feature.multi_target', str, varargs=( - str, File, build.CustomTarget, build.CustomTargetIndex, - build.GeneratedList, build.StructuredSources, build.ExtractedObjects, - build.BuildTarget - )) - @typed_kwargs('feature.multi_target', - KwargInfo( - 'dispatch', ContainerTypeInfo(list, - (FeatureObject, list) - ), - default=[] - ), - KwargInfo( - 'baseline', (NoneType, ContainerTypeInfo(list, FeatureObject)) - ), - KwargInfo('prefix', str, default=''), - KwargInfo('compiler', (NoneType, Compiler)), - allow_unknown=True - ) - def multi_target_method(self, state: 'ModuleState', - args: T.Tuple[str], kwargs: 'TYPE_kwargs' - ) -> T.List[T.Union[T.Dict[str, str], T.Any]]: - config_name = args[0] - sources = args[1] - dispatch = kwargs.pop('dispatch') - baseline = kwargs.pop('baseline') - prefix = kwargs.pop('prefix') - compiler = kwargs.pop('compiler') - if not compiler: - compiler = get_compiler(state) - - info = {} - if baseline is not None: - baseline_features = self.implicit_c(baseline) - cached, baseline = self.test( - state, features=set(baseline), anyfet=True, - compiler=compiler - ) - info['BASELINE'] = baseline - else: - baseline_features = [] - - dispatch_tests = [] - for d in dispatch: - if isinstance(d, FeatureObject): - target = {d,} - is_base_part = d in baseline_features - else: - target = set(d) - is_base_part = all([f in baseline_features for f in d]) - if is_base_part: - # TODO: add log - continue - cached, test_result = self.test( - state, features=target, - compiler=compiler - ) - if not test_result['is_supported']: - continue - target_name = test_result['target_name'] - if target_name in info: - continue - info[target_name] = test_result - dispatch_tests.append(test_result) - - dispatch_calls = [] - for test_result in dispatch_tests: - detect = '&&'.join([ - f'TEST_CB({d})' for d in test_result['detect'] - ]) - if detect: - detect = f'({detect})' - else: - detect = '1' - target_name = test_result['target_name'] - dispatch_calls.append( - f'{prefix}_MTARGETS_EXPAND(' - f'EXEC_CB({detect}, {target_name}, __VA_ARGS__)' - ')' - ) - - config_file = [ - '/* Autogenerated by the Meson features module. */', - '/* Do not edit, your changes will be lost. */', - '', - f'#undef {prefix}_MTARGETS_EXPAND', - f'#define {prefix}_MTARGETS_EXPAND(X) X', - '', - f'#undef {prefix}MTARGETS_CONF_BASELINE', - f'#define {prefix}MTARGETS_CONF_BASELINE(EXEC_CB, ...) ' + ( - f'{prefix}_MTARGETS_EXPAND(EXEC_CB(__VA_ARGS__))' - if baseline is not None - else '' - ), - '', - f'#undef {prefix}MTARGETS_CONF_DISPATCH', - f'#define {prefix}MTARGETS_CONF_DISPATCH(TEST_CB, EXEC_CB, ...) \\', - ' \\\n'.join(dispatch_calls), - '', - ] - - src_dir = state.environment.source_dir - sub_dir = state.subdir - if sub_dir: - src_dir = os.path.join(src_dir, state.subdir) - config_path = os.path.abspath(os.path.join(src_dir, config_name)) - - mlog.log( - "Generating", config_name, 'into path', config_path, - "based on the specifed targets" - ) - os.makedirs(os.path.dirname(config_path), exist_ok=True) - with open(config_path, "w", encoding='utf-8') as cout: - cout.write('\n'.join(config_file)) - - static_libs = [] - if baseline: - static_libs.append(self.gen_target( - state, config_name, sources, - baseline, prefix, True, kwargs - )) - - for test_result in dispatch_tests: - static_libs.append(self.gen_target( - state, config_name, sources, - test_result, prefix, - False, kwargs - )) - return [info, static_libs] - - def gen_target(self, state, config_name, sources, - test_result, prefix, - is_baseline, stlib_kwargs): - target_name = 'baseline' if is_baseline else test_result['target_name'] - args = [f'-D{prefix}HAVE_{df}' for df in test_result['defines']] - args += test_result['args'] - if is_baseline: - args.append(f'-D{prefix}MTARGETS_BASELINE') - else: - args.append(f'-D{prefix}MTARGETS_CURRENT={target_name}') - stlib_kwargs = stlib_kwargs.copy() - stlib_kwargs.update({ - 'sources': sources, - 'c_args': stlib_kwargs.get('c_args', []) + args, - 'cpp_args': stlib_kwargs.get('cpp_args', []) + args - }) - static_lib = state._interpreter.func_static_lib( - None, [config_name + '_' + target_name], - stlib_kwargs - ) - return static_lib - - @typed_pos_args('feature.sort', varargs=FeatureObject, min_varargs=1) - @typed_kwargs('feature.sort', - KwargInfo('reverse', bool, default = False), - ) - def sort_method(self, state: 'ModuleState', - args: T.Tuple[T.List[FeatureObject]], - kwargs: 'TYPE_kwargs' - ) -> T.List[FeatureObject]: - return sorted(args[0], reverse=kwargs['reverse']) - - @typed_pos_args('feature.implicit', varargs=FeatureObject, min_varargs=1) - @noKwargs - def implicit_method(self, state: 'ModuleState', - args: T.Tuple[T.List[FeatureObject]], - kwargs: 'TYPE_kwargs' - ) -> T.List[FeatureObject]: - - features = args[0] - return sorted(self.implicit(features)) - - @typed_pos_args('feature.implicit', varargs=FeatureObject, min_varargs=1) - @noKwargs - def implicit_c_method(self, state: 'ModuleState', - args: T.Tuple[T.List[FeatureObject]], - kwargs: 'TYPE_kwargs' - ) -> T.List[FeatureObject]: - return sorted(self.implicit_c(args[0])) - - @staticmethod - def implicit(features: T.Sequence[FeatureObject]) -> T.Set[FeatureObject]: - implies = set().union(*[f.get_implicit() for f in features]) - return implies - - @staticmethod - def implicit_c(features: T.Sequence[FeatureObject]) -> T.Set[FeatureObject]: - return Module.implicit(features).union(features) - diff --git a/mesonbuild/modules/features/__init__.py b/mesonbuild/modules/features/__init__.py new file mode 100644 index 000000000000..3b56affcfbe3 --- /dev/null +++ b/mesonbuild/modules/features/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2023, NumPy Developers. +# All rights reserved. +# + +import typing as T + +from .module import Module + +if T.TYPE_CHECKING: + from ...interpreter import Interpreter + +def initialize(interpreter: 'Interpreter') -> Module: + return Module() diff --git a/mesonbuild/modules/feature/feature.py b/mesonbuild/modules/features/feature.py similarity index 74% rename from mesonbuild/modules/feature/feature.py rename to mesonbuild/modules/features/feature.py index e269d6231270..f4021484e28e 100644 --- a/mesonbuild/modules/feature/feature.py +++ b/mesonbuild/modules/features/feature.py @@ -4,16 +4,14 @@ import typing as T import re from dataclasses import dataclass, field -from enum import IntFlag, auto from ...mesonlib import File, MesonException from ...interpreter.type_checking import NoneType from ...interpreterbase.decorators import ( noKwargs, noPosargs, KwargInfo, typed_kwargs, typed_pos_args, - ContainerTypeInfo, noArgsFlattening + ContainerTypeInfo ) from .. import ModuleObject -from .utils import test_code, get_compiler if T.TYPE_CHECKING: from typing import TypedDict @@ -50,9 +48,6 @@ class ConflictAttr: ) mjoin: str = field(default='', hash=False, compare=False) - def __str__(self) -> str: - return self.val - def copy(self) -> 'ConflictAttr': return ConflictAttr(**self.__dict__) @@ -71,14 +66,12 @@ def to_dict(self) -> T.Dict[str, str]: class KwargConfilctAttr(KwargInfo): def __init__(self, func_name: str, opt_name: str, default: T.Any = None): - types = [ - str, ContainerTypeInfo(dict, str), + types = ( + NoneType, str, ContainerTypeInfo(dict, str), ContainerTypeInfo(list, (dict, str)) - ] - if default is None: - types += [NoneType] + ) super().__init__( - opt_name, tuple(types), + opt_name, types, convertor = lambda values: self.convert( func_name, opt_name, values ), @@ -152,29 +145,6 @@ class FeatureUpdateKwArgs(FeatureKwArgs): interest: NotRequired[int] class FeatureObject(ModuleObject): - """ - A data class that represents the feature. - - A feature is a unit of work that can be developed, tested, and deployed independently. - - Attributes: - name: The name of the feature. - interest: The interest level of the feature. - It used for sorting and to determine succor features. - implies: A set of features objects that are implied by this feature. - (Optional) - This means that if this feature is enabled, - then the implied features will also be enabled. - If any of the implied features is not supported by the platform - or the compiler this feature will also considerd not supported. - group: A list of - - detect: A list of strings that identify the methods that can be used to detect whether the feature is supported. - args: A list of strings that identify the arguments that are required to enable this feature. - test_code: A string or a file object that contains the test code for this feature. - extra_tests: A dictionary that maps from the name of a test to the test code for that test. - disable: A string that specifies why this feature is disabled. - """ name: str interest: int implies: T.Set['FeatureObject'] @@ -191,8 +161,8 @@ def __init__(self, state: 'ModuleState', super().__init__() - @typed_pos_args('feature.new', str, int) - @typed_kwargs('feature.new', + @typed_pos_args('features.new', str, int) + @typed_kwargs('features.new', KwargInfo( 'implies', (FeatureObject, ContainerTypeInfo(list, FeatureObject)), @@ -202,8 +172,8 @@ def __init__(self, state: 'ModuleState', 'group', (str, ContainerTypeInfo(list, str)), default=[], listify=True ), - KwargConfilctAttr('feature.new', 'detect', default=[]), - KwargConfilctAttr('feature.new', 'args', default=[]), + KwargConfilctAttr('features.new', 'detect', default=[]), + KwargConfilctAttr('features.new', 'args', default=[]), KwargInfo('test_code', (str, File), default=''), KwargInfo( 'extra_tests', (ContainerTypeInfo(dict, (str, File))), @@ -238,7 +208,7 @@ def init_attrs(state: 'ModuleState', def update_method(self, state: 'ModuleState', args: T.List['TYPE_var'], kwargs: 'TYPE_kwargs') -> 'FeatureObject': @noPosargs - @typed_kwargs('feature.update', + @typed_kwargs('features.FeatureObject.update', KwargInfo('name', (NoneType, str)), KwargInfo('interest', (NoneType, int)), KwargInfo( @@ -252,8 +222,8 @@ def update_method(self, state: 'ModuleState', args: T.List['TYPE_var'], 'group', (NoneType, str, ContainerTypeInfo(list, str)), listify=True ), - KwargConfilctAttr('feature.update', 'detect'), - KwargConfilctAttr('feature.update', 'args'), + KwargConfilctAttr('features.FeatureObject.update', 'detect'), + KwargConfilctAttr('features.FeatureObject.update', 'args'), KwargInfo('test_code', (NoneType, str, File)), KwargInfo( 'extra_tests', ( @@ -273,27 +243,28 @@ def update(state: 'ModuleState', args: T.List['TYPE_var'], return self @noKwargs - @typed_pos_args('feature.get', str) + @typed_pos_args('features.FeatureObject.get', str) def get_method(self, state: 'ModuleState', args: T.Tuple[str], kwargs: 'TYPE_kwargs') -> 'TYPE_var': impl_lst = lambda lst: [v.to_dict() for v in lst] noconv = lambda v: v - dfunc = dict( - name = noconv, - interest = noconv, - group = noconv, - implies = lambda v: [fet.name for fet in sorted(v)], - detect = impl_lst, - args = impl_lst, - test_code = noconv, - extra_tests = noconv, - disable = noconv - ) - cfunc = dfunc.get(args[0]) + dfunc = { + 'name': noconv, + 'interest': noconv, + 'group': noconv, + 'implies': lambda v: [fet.name for fet in sorted(v)], + 'detect': impl_lst, + 'args': impl_lst, + 'test_code': noconv, + 'extra_tests': noconv, + 'disable': noconv + } + cfunc: T.Optional[T.Callable[[str], 'TYPE_var']] = dfunc.get(args[0]) if cfunc is None: raise MesonException(f'Key {args[0]!r} is not in the feature.') - return cfunc(getattr(self, args[0])) + val = getattr(self, args[0]) + return cfunc(val) def get_implicit(self, _caller: T.Set['FeatureObject'] = None ) -> T.Set['FeatureObject']: @@ -306,8 +277,61 @@ def get_implicit(self, _caller: T.Set['FeatureObject'] = None ret = ret.union(sub_fet.get_implicit(_caller)) return ret + @staticmethod + def get_implicit_multi(features: T.Iterable['FeatureObject']) -> T.Set['FeatureObject']: + implies = set().union(*[f.get_implicit() for f in features]) + return implies + + @staticmethod + def get_implicit_combine_multi(features: T.Iterable['FeatureObject']) -> T.Set['FeatureObject']: + return FeatureObject.get_implicit_multi(features).union(features) + + @staticmethod + def sorted_multi(features: T.Iterable[T.Union['FeatureObject', T.Iterable['FeatureObject']]], + reverse: bool = False + ) -> T.List[T.Union['FeatureObject', T.Iterable['FeatureObject']]]: + def sort_cb(k: T.Union[FeatureObject, T.Iterable[FeatureObject]]) -> int: + if isinstance(k, FeatureObject): + return k.interest + # keep prevalent features and erase any implied features + implied_features = FeatureObject.get_implicit_multi(k) + prevalent_features = set(k).difference(implied_features) + if len(prevalent_features) == 0: + # It happens when all features imply each other. + # Set the highest interested feature + return sorted(k)[-1].interest + # multiple features + rank = max(f.interest for f in prevalent_features) + # FIXME: that's not a safe way to increase the rank for + # multi features this why this function isn't considerd + # accurate. + rank += len(prevalent_features) -1 + return rank + return sorted(features, reverse=reverse, key=sort_cb) + + @staticmethod + def features_names(features: T.Iterable[T.Union['FeatureObject', T.Iterable['FeatureObject']]] + ) -> T.List[T.Union[str, T.List[str]]]: + return [ + fet.name if isinstance(fet, FeatureObject) + else [f.name for f in fet] + for fet in features + ] + + def __repr__(self) -> str: + args = ', '.join([ + f'{attr} = {str(getattr(self, attr))}' + for attr in [ + 'group', 'implies', + 'detect', 'args', + 'test_code', 'extra_tests', + 'disable' + ] + ]) + return f'FeatureObject({self.name}, {self.interest}, {args})' + def __hash__(self) -> int: - return hash(str(id(self)) + self.name) + return hash(self.name) def __eq__(self, robj: object) -> bool: if not isinstance(robj, FeatureObject): @@ -329,4 +353,3 @@ def __gt__(self, robj: object) -> T.Any: def __ge__(self, robj: object) -> T.Any: return robj <= self - diff --git a/mesonbuild/modules/features/module.py b/mesonbuild/modules/features/module.py new file mode 100644 index 000000000000..8a07ab04d383 --- /dev/null +++ b/mesonbuild/modules/features/module.py @@ -0,0 +1,714 @@ +# Copyright (c) 2023, NumPy Developers. +# All rights reserved. + +import typing as T +import os + +from ... import mlog, build +from ...compilers import Compiler +from ...mesonlib import File, MesonException +from ...interpreter.type_checking import NoneType +from ...interpreterbase.decorators import ( + noKwargs, KwargInfo, typed_kwargs, typed_pos_args, + ContainerTypeInfo, permittedKwargs +) +from .. import ModuleInfo, NewExtensionModule, ModuleObject +from .feature import FeatureObject, ConflictAttr +from .utils import test_code, get_compiler, generate_hash + +if T.TYPE_CHECKING: + from typing import TypedDict + from ...interpreterbase import TYPE_var, TYPE_kwargs + from .. import ModuleState + from .feature import FeatureKwArgs + + class TestKwArgs(TypedDict): + compiler: T.Optional[Compiler] + force_args: T.Optional[T.List[str]] + anyfet: bool + cached: bool + + class TestResultKwArgs(TypedDict): + target_name: str + prevalent_features: T.List[str] + features: T.List[str] + args: T.List[str] + detect: T.List[str] + defines: T.List[str] + undefines: T.List[str] + is_supported: bool + is_disabled: bool + fail_reason: str + +class TargetsObject(ModuleObject): + def __init__(self) -> None: + super().__init__() + self._targets: T.Dict[ + T.Union[FeatureObject, T.Tuple[FeatureObject, ...]], + T.List[build.StaticLibrary] + ] = {} + self._baseline: T.List[build.StaticLibrary] = [] + self.methods.update({ + 'static_lib': self.static_lib_method, + 'extend': self.extend_method + }) + + def extend_method(self, state: 'ModuleState', + args: T.List['TYPE_var'], + kwargs: 'TYPE_kwargs') -> 'TargetsObject': + + @typed_pos_args('feature.TargetsObject.extend', TargetsObject) + @noKwargs + def test_args(state: 'ModuleState', + args: T.Tuple[TargetsObject], + kwargs: 'TYPE_kwargs') -> TargetsObject: + return args[0] + robj: TargetsObject = test_args(state, args, kwargs) + self._baseline.extend(robj._baseline) + for features, robj_targets in robj._targets.items(): + targets: T.List[build.StaticLibrary] = self._targets.setdefault(features, []) + targets += robj_targets + return self + + @typed_pos_args('features.TargetsObject.static_lib', str) + @noKwargs + def static_lib_method(self, state: 'ModuleState', args: T.Tuple[str], + kwargs: 'TYPE_kwargs' + ) -> T.Any: + # The linking order must be based on the lowest interested features, + # to ensures that the linker prioritizes any duplicate weak global symbols + # of the lowest interested features over the highest ones, + # starting with the baseline to avoid any possible crashes due + # to any involved optimizations that may generated based + # on the highest interested features. + link_whole = [] + self._baseline + tcast = T.Union[FeatureObject, T.Tuple[FeatureObject, ...]] + for features in FeatureObject.sorted_multi(self._targets.keys()): + link_whole += self._targets[T.cast(tcast, features)] + if not link_whole: + return [] + static_lib = state._interpreter.func_static_lib( + None, [args[0]], { + 'link_whole': link_whole + } + ) + return static_lib + + def add_baseline_target(self, target: build.StaticLibrary) -> None: + self._baseline.append(target) + + def add_target(self, features: T.Union[FeatureObject, T.List[FeatureObject]], + target: build.StaticLibrary) -> None: + tfeatures = ( + features if isinstance(features, FeatureObject) + else tuple(sorted(features)) + ) + targets: T.List[build.StaticLibrary] = self._targets.setdefault( + tfeatures, T.cast(T.List[build.StaticLibrary], [])) # type: ignore + targets.append(target) + +class Module(NewExtensionModule): + INFO = ModuleInfo('features', '0.1.0') + def __init__(self) -> None: + super().__init__() + self.methods.update({ + 'new': self.new_method, + 'test': self.test_method, + 'implicit': self.implicit_method, + 'implicit_c': self.implicit_c_method, + 'sort': self.sort_method, + 'multi_targets': self.multi_targets_method, + }) + + def new_method(self, state: 'ModuleState', + args: T.List['TYPE_var'], + kwargs: 'TYPE_kwargs') -> FeatureObject: + return FeatureObject(state, args, kwargs) + + def _cache_dict(self, state: 'ModuleState' + ) -> T.Dict[str, 'TestResultKwArgs']: + coredata = state.environment.coredata + attr_name = 'module_features_cache' + if not hasattr(coredata, attr_name): + setattr(coredata, attr_name, {}) + return getattr(coredata, attr_name, {}) + + def _get_cache(self, state: 'ModuleState', key: str + ) -> T.Optional['TestResultKwArgs']: + return self._cache_dict(state).get(key) + + def _set_cache(self, state: 'ModuleState', key: str, + val: 'TestResultKwArgs') -> None: + self._cache_dict(state)[key] = val + + @typed_pos_args('features.test', varargs=FeatureObject, min_varargs=1) + @typed_kwargs('features.test', + KwargInfo('compiler', (NoneType, Compiler)), + KwargInfo('anyfet', bool, default = False), + KwargInfo('cached', bool, default = True), + KwargInfo( + 'force_args', (NoneType, str, ContainerTypeInfo(list, str)), + listify=True + ), + ) + def test_method(self, state: 'ModuleState', + args: T.Tuple[T.List[FeatureObject]], + kwargs: 'TestKwArgs' + ) -> T.List[T.Union[bool, 'TestResultKwArgs']]: + + features = args[0] + features_set = set(features) + anyfet = kwargs['anyfet'] + cached = kwargs['cached'] + compiler = kwargs.get('compiler') + if not compiler: + compiler = get_compiler(state) + + force_args = kwargs['force_args'] + if force_args is not None: + # removes in empty strings + force_args = [a for a in force_args if a] + + test_cached, test_result = self.cached_test( + state, features=features_set, + compiler=compiler, + anyfet=anyfet, + cached=cached, + force_args=force_args + ) + if not test_result['is_supported']: + if test_result['is_disabled']: + label = mlog.yellow('disabled') + else: + label = mlog.yellow('Unsupported') + else: + label = mlog.green('Supported') + if anyfet: + unsupported = ' '.join([ + fet.name for fet in sorted(features_set) + if fet.name not in test_result['features'] + ]) + if unsupported: + label = mlog.green(f'Parial support, missing({unsupported})') + + features_names = ' '.join([f.name for f in features]) + log_prefix = f'Test features "{mlog.bold(features_names)}" :' + cached_msg = f'({mlog.blue("cached")})' if test_cached else '' + if not test_result['is_supported']: + mlog.log(log_prefix, label, 'due to', test_result['fail_reason']) + else: + mlog.log(log_prefix, label, cached_msg) + return [test_result['is_supported'], test_result] + + def cached_test(self, state: 'ModuleState', + features: T.Set[FeatureObject], + compiler: 'Compiler', + force_args: T.Optional[T.List[str]], + anyfet: bool, cached: bool, + _caller: T.Optional[T.Set[FeatureObject]] = None + ) -> T.Tuple[bool, 'TestResultKwArgs']: + + if cached: + test_hash = generate_hash( + sorted(features), compiler, + anyfet, force_args + ) + test_result = self._get_cache(state, test_hash) + if test_result is not None: + return True, test_result + + if anyfet: + test_func = self.test_any + else: + test_func = self.test + + test_result = test_func( + state, features=features, + compiler=compiler, + force_args=force_args, + cached=cached, + _caller=_caller + ) + if cached: + self._set_cache(state, test_hash, test_result) + return False, test_result + + def test_any(self, state: 'ModuleState', features: T.Set[FeatureObject], + compiler: 'Compiler', + force_args: T.Optional[T.List[str]], + cached: bool, + # dummy no need for recrusive guard + _caller: T.Optional[T.Set[FeatureObject]] = None, + ) -> 'TestResultKwArgs': + + _, test_any_result = self.cached_test( + state, features=features, + compiler=compiler, + anyfet=False, + cached=cached, + force_args=force_args, + ) + if test_any_result['is_supported']: + return test_any_result + + all_features = sorted(FeatureObject.get_implicit_combine_multi(features)) + features_any = set() + for fet in all_features: + _, test_any_result = self.cached_test( + state, features={fet,}, + compiler=compiler, + cached=cached, + anyfet=False, + force_args=force_args, + ) + if test_any_result['is_supported']: + features_any.add(fet) + + _, test_any_result = self.cached_test( + state, features=features_any, + compiler=compiler, + cached=cached, + anyfet=False, + force_args=force_args, + ) + return test_any_result + + def test(self, state: 'ModuleState', features: T.Set[FeatureObject], + compiler: 'Compiler', + force_args: T.Optional[T.List[str]] = None, + cached: bool = True, + _caller: T.Optional[T.Set[FeatureObject]] = None + ) -> 'TestResultKwArgs': + + implied_features = FeatureObject.get_implicit_multi(features) + all_features = sorted(implied_features.union(features)) + # For multiple features, it important to erase any features + # implied by another to avoid duplicate testing since + # implied features already tested also we use this set to genrate + # unque target name that can be used for multiple targets + # build. + prevalent_features = sorted(features.difference(implied_features)) + if len(prevalent_features) == 0: + # It happens when all features imply each other. + # Set the highest interested feature + prevalent_features = sorted(features)[-1:] + + prevalent_names = [fet.name for fet in prevalent_features] + # prepare the result dict + test_result: 'TestResultKwArgs' = { + 'target_name': '__'.join(prevalent_names), + 'prevalent_features': prevalent_names, + 'features': [fet.name for fet in all_features], + 'args': [], + 'detect': [], + 'defines': [], + 'undefines': [], + 'is_supported': True, + 'is_disabled': False, + 'fail_reason': '', + } + def fail_result(fail_reason: str, is_disabled: bool = False + ) -> 'TestResultKwArgs': + test_result.update({ + 'features': [], + 'args': [], + 'detect': [], + 'defines': [], + 'undefines': [], + 'is_supported': False, + 'is_disabled': is_disabled, + 'fail_reason': fail_reason, + }) + return test_result + + # test any of prevalent features wither they disabled or not + for fet in prevalent_features: + if fet.disable: + return fail_result( + f'{fet.name} is disabled due to "{fet.disable}"', + True + ) + + # since we allows features to imply each other + # items of `features` may part of `implied_features` + if _caller is None: + _caller = set() + _caller = _caller.union(prevalent_features) + predecessor_features = implied_features.difference(_caller) + for fet in sorted(predecessor_features): + _, pred_result = self.cached_test( + state, features={fet,}, + compiler=compiler, + cached=cached, + anyfet=False, + force_args=force_args, + _caller=_caller, + ) + if not pred_result['is_supported']: + reason = f'Implied feature "{fet.name}" ' + pred_disabled = pred_result['is_disabled'] + if pred_disabled: + fail_reason = reason + 'is disabled' + else: + fail_reason = reason + 'is not supported' + return fail_result(fail_reason, pred_disabled) + + for k in ['defines', 'undefines']: + def_values = test_result[k] # type: ignore + pred_values = pred_result[k] # type: ignore + def_values += [v for v in pred_values if v not in def_values] + + # Sort based on the lowest interest to deal with conflict attributes + # when combine all attributes togathers + conflict_attrs = ['detect'] + if force_args is None: + conflict_attrs += ['args'] + else: + test_result['args'] = force_args + + for fet in all_features: + for attr in conflict_attrs: + values: T.List[ConflictAttr] = getattr(fet, attr) + accumulate_values = test_result[attr] # type: ignore + for conflict in values: + if not conflict.match: + accumulate_values.append(conflict.val) + continue + conflict_vals: T.List[str] = [] + # select the acc items based on the match + new_acc: T.List[str] = [] + for acc in accumulate_values: + # not affected by the match so we keep it + if not conflict.match.match(acc): + new_acc.append(acc) + continue + # no filter so we totaly escape it + if not conflict.mfilter: + continue + filter_val = conflict.mfilter.findall(acc) + filter_val = [ + conflict.mjoin.join([i for i in val if i]) + if isinstance(val, tuple) else val + for val in filter_val if val + ] + # no filter match so we totaly escape it + if not filter_val: + continue + conflict_vals.append(conflict.mjoin.join(filter_val)) + new_acc.append(conflict.val + conflict.mjoin.join(conflict_vals)) + test_result[attr] = new_acc # type: ignore + + test_args = compiler.has_multi_arguments + args = test_result['args'] + if args: + supported_args, test_cached = test_args(args, state.environment) + if not supported_args: + return fail_result( + f'Arguments "{", ".join(args)}" are not supported' + ) + + for fet in prevalent_features: + if fet.test_code: + _, tested_code, _ = test_code( + state, compiler, args, fet.test_code + ) + if not tested_code: + return fail_result( + f'Compiler fails against the test code of "{fet.name}"' + ) + + test_result['defines'] += [fet.name] + fet.group + for extra_name, extra_test in fet.extra_tests.items(): + _, tested_code, _ = test_code( + state, compiler, args, extra_test + ) + k = 'defines' if tested_code else 'undefines' + test_result[k].append(extra_name) # type: ignore + return test_result + + @permittedKwargs(build.known_stlib_kwargs | { + 'dispatch', 'baseline', 'prefix', 'cached', 'keep_sort' + }) + @typed_pos_args('features.multi_targets', str, min_varargs=1, varargs=( + str, File, build.CustomTarget, build.CustomTargetIndex, + build.GeneratedList, build.StructuredSources, build.ExtractedObjects, + build.BuildTarget + )) + @typed_kwargs('features.multi_targets', + KwargInfo( + 'dispatch', ( + ContainerTypeInfo(list, (FeatureObject, list)), + ), + default=[] + ), + KwargInfo( + 'baseline', ( + NoneType, + ContainerTypeInfo(list, FeatureObject) + ) + ), + KwargInfo('prefix', str, default=''), + KwargInfo('compiler', (NoneType, Compiler)), + KwargInfo('cached', bool, default = True), + KwargInfo('keep_sort', bool, default = False), + allow_unknown=True + ) + def multi_targets_method(self, state: 'ModuleState', + args: T.Tuple[str], kwargs: 'TYPE_kwargs' + ) -> TargetsObject: + config_name = args[0] + sources = args[1] # type: ignore + dispatch: T.List[T.Union[FeatureObject, T.List[FeatureObject]]] = ( + kwargs.pop('dispatch') # type: ignore + ) + baseline: T.Optional[T.List[FeatureObject]] = ( + kwargs.pop('baseline') # type: ignore + ) + prefix: str = kwargs.pop('prefix') # type: ignore + cached: bool = kwargs.pop('cached') # type: ignore + compiler: T.Optional[Compiler] = kwargs.pop('compiler') # type: ignore + if not compiler: + compiler = get_compiler(state) + + baseline_features : T.Set[FeatureObject] = set() + has_baseline = baseline is not None + if has_baseline: + baseline_features = FeatureObject.get_implicit_combine_multi(baseline) + _, baseline_test_result = self.cached_test( + state, features=set(baseline), + anyfet=True, cached=cached, + compiler=compiler, + force_args=None + ) + + enabled_targets_names: T.List[str] = [] + enabled_targets_features: T.List[T.Union[ + FeatureObject, T.List[FeatureObject] + ]] = [] + enabled_targets_tests: T.List['TestResultKwArgs'] = [] + skipped_targets: T.List[T.Tuple[ + T.Union[FeatureObject, T.List[FeatureObject]], str + ]] = [] + for d in dispatch: + if isinstance(d, FeatureObject): + target = {d,} + is_base_part = d in baseline_features + else: + target = set(d) + is_base_part = all(f in baseline_features for f in d) + + if is_base_part: + skipped_targets.append((d, "part of baseline features")) + continue + _, test_result = self.cached_test( + state=state, features=target, + anyfet=False, cached=cached, + compiler=compiler, + force_args=None + ) + if not test_result['is_supported']: + skipped_targets.append( + (d, test_result['fail_reason']) + ) + continue + target_name = test_result['target_name'] + if target_name in enabled_targets_names: + skipped_targets.append(( + d, f'Dublicate target name "{target_name}"' + )) + continue + enabled_targets_names.append(target_name) + enabled_targets_features.append(d) + enabled_targets_tests.append(test_result) + + if not kwargs.pop('keep_sort'): + enabled_targets_sorted = FeatureObject.sorted_multi(enabled_targets_features, reverse=True) + if enabled_targets_features != enabled_targets_sorted: + log_targets = FeatureObject.features_names(enabled_targets_features) + log_targets_sorted = FeatureObject.features_names(enabled_targets_sorted) + raise MesonException( + 'The enabled dispatch features should be sorted based on the highest interest:\n' + f'Expected: {log_targets_sorted}\n' + f'Got: {log_targets}\n' + 'Note: This validation may not be accurate when dealing with multi-features ' + 'per single target.\n' + 'You can keep the current sort and bypass this validation by passing ' + 'the argument "keep_sort: true".' + ) + + config_path = self.gen_config( + state, + config_name=config_name, + targets=enabled_targets_tests, + prefix=prefix, + has_baseline=has_baseline + ) + mtargets_obj = TargetsObject() + if has_baseline: + mtargets_obj.add_baseline_target( + self.gen_target( + state=state, config_name=config_name, + sources=sources, test_result=baseline_test_result, + prefix=prefix, is_baseline=True, + stlib_kwargs=kwargs + ) + ) + for features_objects, target_test in zip(enabled_targets_features, enabled_targets_tests): + static_lib = self.gen_target( + state=state, config_name=config_name, + sources=sources, test_result=target_test, + prefix=prefix, is_baseline=False, + stlib_kwargs=kwargs + ) + mtargets_obj.add_target(features_objects, static_lib) + + skipped_targets_info: T.List[str] = [] + skipped_tab = ' '*4 + for skipped, reason in skipped_targets: + name = ', '.join( + [skipped.name] if isinstance(skipped, FeatureObject) + else [fet.name for fet in skipped] + ) + skipped_targets_info.append(f'{skipped_tab}"{name}": "{reason}"') + + target_info: T.Callable[[str, 'TestResultKwArgs'], str] = lambda target_name, test_result: ( + f'{skipped_tab}"{target_name}":\n' + '\n'.join([ + f'{skipped_tab*2}"{k}": {v}' + for k, v in test_result.items() + ]) + ) + enabled_targets_info: T.List[str] = [ + target_info(test_result['target_name'], test_result) + for test_result in enabled_targets_tests + ] + if has_baseline: + enabled_targets_info.append(target_info( + f'baseline({baseline_test_result["target_name"]})', + baseline_test_result + )) + enabled_targets_names += ['baseline'] + + mlog.log( + f'Generating multi-targets for "{mlog.bold(config_name)}"', + '\n Enabled targets:', + mlog.green(', '.join(enabled_targets_names)) + ) + mlog.debug( + f'Generating multi-targets for "{config_name}"', + '\n Config path:', config_path, + '\n Enabled targets:', + '\n'+'\n'.join(enabled_targets_info), + '\n Skipped targets:', + '\n'+'\n'.join(skipped_targets_info), + '\n' + ) + return mtargets_obj + + def gen_target(self, state: 'ModuleState', config_name: str, + sources: T.List[T.Union[ + str, File, build.CustomTarget, build.CustomTargetIndex, + build.GeneratedList, build.StructuredSources, build.ExtractedObjects, + build.BuildTarget + ]], + test_result: 'TestResultKwArgs', + prefix: str, is_baseline: bool, + stlib_kwargs: T.Dict[str, T.Any] + ) -> build.StaticLibrary: + + target_name = 'baseline' if is_baseline else test_result['target_name'] + args = [f'-D{prefix}HAVE_{df}' for df in test_result['defines']] + args += test_result['args'] + if is_baseline: + args.append(f'-D{prefix}MTARGETS_BASELINE') + else: + args.append(f'-D{prefix}MTARGETS_CURRENT={target_name}') + stlib_kwargs = stlib_kwargs.copy() + stlib_kwargs.update({ + 'sources': sources, + 'c_args': stlib_kwargs.get('c_args', []) + args, + 'cpp_args': stlib_kwargs.get('cpp_args', []) + args + }) + static_lib: build.StaticLibrary = state._interpreter.func_static_lib( + None, [config_name + '_' + target_name], + stlib_kwargs + ) + return static_lib + + def gen_config(self, state: 'ModuleState', config_name: str, + targets: T.List['TestResultKwArgs'], + prefix: str, has_baseline: bool + ) -> str: + + dispatch_calls: T.List[str] = [] + for test in targets: + c_detect = '&&'.join([ + f'TEST_CB({d})' for d in test['detect'] + ]) + if c_detect: + c_detect = f'({c_detect})' + else: + c_detect = '1' + dispatch_calls.append( + f'{prefix}_MTARGETS_EXPAND(' + f'EXEC_CB({c_detect}, {test["target_name"]}, __VA_ARGS__)' + ')' + ) + + config_file = [ + '/* Autogenerated by the Meson features module. */', + '/* Do not edit, your changes will be lost. */', + '', + f'#undef {prefix}_MTARGETS_EXPAND', + f'#define {prefix}_MTARGETS_EXPAND(X) X', + '', + f'#undef {prefix}MTARGETS_CONF_BASELINE', + f'#define {prefix}MTARGETS_CONF_BASELINE(EXEC_CB, ...) ' + ( + f'{prefix}_MTARGETS_EXPAND(EXEC_CB(__VA_ARGS__))' + if has_baseline else '' + ), + '', + f'#undef {prefix}MTARGETS_CONF_DISPATCH', + f'#define {prefix}MTARGETS_CONF_DISPATCH(TEST_CB, EXEC_CB, ...) \\', + ' \\\n'.join(dispatch_calls), + '', + ] + + build_dir = state.environment.build_dir + sub_dir = state.subdir + if sub_dir: + build_dir = os.path.join(build_dir, sub_dir) + config_path = os.path.abspath(os.path.join(build_dir, config_name)) + + os.makedirs(os.path.dirname(config_path), exist_ok=True) + with open(config_path, "w", encoding='utf-8') as cout: + cout.write('\n'.join(config_file)) + + return config_path + + @typed_pos_args('features.sort', varargs=FeatureObject, min_varargs=1) + @typed_kwargs('features.sort', + KwargInfo('reverse', bool, default = False), + ) + def sort_method(self, state: 'ModuleState', + args: T.Tuple[T.List[FeatureObject]], + kwargs: T.Dict[str, bool] + ) -> T.List[FeatureObject]: + return sorted(args[0], reverse=kwargs['reverse']) + + @typed_pos_args('features.implicit', varargs=FeatureObject, min_varargs=1) + @noKwargs + def implicit_method(self, state: 'ModuleState', + args: T.Tuple[T.List[FeatureObject]], + kwargs: 'TYPE_kwargs' + ) -> T.List[FeatureObject]: + + features = args[0] + return sorted(FeatureObject.get_implicit_multi(features)) + + @typed_pos_args('features.implicit', varargs=FeatureObject, min_varargs=1) + @noKwargs + def implicit_c_method(self, state: 'ModuleState', + args: T.Tuple[T.List[FeatureObject]], + kwargs: 'TYPE_kwargs' + ) -> T.List[FeatureObject]: + return sorted(FeatureObject.get_implicit_combine_multi(args[0])) diff --git a/mesonbuild/modules/feature/utils.py b/mesonbuild/modules/features/utils.py similarity index 83% rename from mesonbuild/modules/feature/utils.py rename to mesonbuild/modules/features/utils.py index b681e96c169a..e7f82935d553 100644 --- a/mesonbuild/modules/feature/utils.py +++ b/mesonbuild/modules/features/utils.py @@ -2,6 +2,7 @@ # All rights reserved. import typing as T +import hashlib from ...mesonlib import MesonException, MachineChoice if T.TYPE_CHECKING: @@ -32,3 +33,9 @@ def test_code(state: 'ModuleState', compiler: 'Compiler', ) as p: return p.cached, p.returncode == 0, p.stderr +def generate_hash(*args: T.Any) -> str: + hasher = hashlib.sha1() + test: T.List[bytes] = [] + for a in args: + hasher.update(bytes(str(a), encoding='utf-8')) + return hasher.hexdigest() diff --git a/run_mypy.py b/run_mypy.py index 2ef61b10b627..efc8498f0339 100755 --- a/run_mypy.py +++ b/run_mypy.py @@ -62,7 +62,7 @@ 'mesonbuild/modules/sourceset.py', 'mesonbuild/modules/wayland.py', 'mesonbuild/modules/windows.py', - 'mesonbuild/modules/feature/', + 'mesonbuild/modules/features/', 'mesonbuild/mparser.py', 'mesonbuild/msetup.py', 'mesonbuild/mtest.py', diff --git a/run_unittests.py b/run_unittests.py index 4dd674d346e1..23e40b475659 100755 --- a/run_unittests.py +++ b/run_unittests.py @@ -50,6 +50,7 @@ from unittests.subprojectscommandtests import SubprojectsCommandTests from unittests.windowstests import WindowsTests from unittests.platformagnostictests import PlatformAgnosticTests +from unittests.featurestests import FeaturesTests def unset_envs(): # For unit tests we must fully control all command lines @@ -119,7 +120,7 @@ def main(): 'TAPParserTests', 'SubprojectsCommandTests', 'PlatformAgnosticTests', 'LinuxlikeTests', 'LinuxCrossArmTests', 'LinuxCrossMingwTests', - 'WindowsTests', 'DarwinTests'] + 'WindowsTests', 'DarwinTests', 'FeaturesTests'] try: import pytest # noqa: F401 diff --git a/test cases/features/1 baseline/baseline.c b/test cases/features/1 baseline/baseline.c new file mode 100644 index 000000000000..c208e9bcc1ac --- /dev/null +++ b/test cases/features/1 baseline/baseline.c @@ -0,0 +1,77 @@ +// the headers files of enabled CPU features +#ifdef HAVE_SSE + #include +#endif +#ifdef HAVE_SSE2 + #include +#endif +#ifdef HAVE_SSE3 + #include +#endif +#ifdef HAVE_SSSE3 + #include +#endif +#ifdef HAVE_SSE41 + #include +#endif +#ifdef HAVE_NEON + #include +#endif + +int main() { +#if defined( __i386__ ) || defined(i386) || defined(_M_IX86) || \ + defined(__x86_64__) || defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) + #ifndef HAVE_SSE + #error "expected SSE to be enabled" + #endif + #ifndef HAVE_SSE2 + #error "expected SSE2 to be enabled" + #endif + #ifndef HAVE_SSE3 + #error "expected SSE3 to be enabled" + #endif +#else + #ifdef HAVE_SSE + #error "expected SSE to be disabled" + #endif + #ifdef HAVE_SSE2 + #error "expected SSE2 to be disabled" + #endif + #ifdef HAVE_SSE3 + #error "expected SSE3 to be disabled" + #endif +#endif + +#if defined(__arm__) + #ifndef HAVE_NEON + #error "expected NEON to be enabled" + #endif +#else + #ifdef HAVE_NEON + #error "expected NEON to be disabled" + #endif +#endif + +#if defined(__aarch64__) || defined(_M_ARM64) + #ifndef HAVE_NEON_FP16 + #error "expected NEON_FP16 to be enabled" + #endif + #ifndef HAVE_NEON_VFPV4 + #error "expected NEON_VFPV4 to be enabled" + #endif + #ifndef HAVE_ASIMD + #error "expected ASIMD to be enabled" + #endif +#else + #ifdef HAVE_NEON_FP16 + #error "expected NEON_FP16 to be disabled" + #endif + #ifdef HAVE_NEON_VFPV4 + #error "expected NEON_VFPV4 to be disabled" + #endif + #ifdef HAVE_ASIMD + #error "expected ASIMD to be disabled" + #endif +#endif + return 0; +} diff --git a/test cases/features/1 baseline/init_features b/test cases/features/1 baseline/init_features new file mode 120000 index 000000000000..d52da150db76 --- /dev/null +++ b/test cases/features/1 baseline/init_features @@ -0,0 +1 @@ +../init_features \ No newline at end of file diff --git a/test cases/features/1 baseline/meson.build b/test cases/features/1 baseline/meson.build new file mode 100644 index 000000000000..71c4299cf3d4 --- /dev/null +++ b/test cases/features/1 baseline/meson.build @@ -0,0 +1,14 @@ +project('baseline', 'c') +subdir('init_features') + +baseline = mod_features.test(SSE3, NEON, anyfet: true) +message(baseline) +baseline_args = baseline[1]['args'] +foreach def : baseline[1]['defines'] + baseline_args += ['-DHAVE_' + def] +endforeach +add_project_arguments(baseline_args, language: ['c', 'cpp']) + +exe = executable('baseline', 'baseline.c') +test('baseline', exe) + diff --git a/test cases/features/2 multi_targets/dispatch.h b/test cases/features/2 multi_targets/dispatch.h new file mode 100644 index 000000000000..d1e00db0f164 --- /dev/null +++ b/test cases/features/2 multi_targets/dispatch.h @@ -0,0 +1,90 @@ +#ifndef DISPATCH_H_ +#define DISPATCH_H_ + +// the headers files of enabled CPU features +#ifdef HAVE_SSE + #include +#endif +#ifdef HAVE_SSE2 + #include +#endif +#ifdef HAVE_SSE3 + #include +#endif +#ifdef HAVE_SSSE3 + #include +#endif +#ifdef HAVE_SSE41 + #include +#endif +#ifdef HAVE_NEON + #include +#endif + +#if defined( __i386__ ) || defined(i386) || defined(_M_IX86) || \ + defined(__x86_64__) || defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) + #define TEST_X86 +#elif defined(__aarch64__) || defined(_M_ARM64) + #define TEST_ARM64 +#elif defined(__arm__) + #define TEST_ARM +#endif + +enum { + CPU_SSE = 1, + CPU_SSE2, + CPU_SSE3, + CPU_SSSE3, + CPU_SSE41, + CPU_NEON, + CPU_NEON_FP16, + CPU_NEON_VFPV4, + CPU_ASIMD +}; +int cpu_has(int feature_id); +#define CPU_TEST(FEATURE_NAME) cpu_has(CPU_##FEATURE_NAME) +#define CPU_TEST_DUMMY(FEATURE_NAME) + +#define EXPAND(X) X +#define CAT__(A, B) A ## B +#define CAT_(A, B) CAT__(A, B) +#define CAT(A, B) CAT_(A, B) +#define STRINGIFY(x) #x +#define TOSTRING(x) STRINGIFY(x) + +#ifdef MTARGETS_CURRENT + #define DISPATCH_CURRENT(X) CAT(CAT(X,_), MTARGETS_CURRENT) +#else + // baseline + #define DISPATCH_CURRENT(X) X +#endif + +#define DISPATCH_DECLARE(...) \ + MTARGETS_CONF_DISPATCH(CPU_TEST_DUMMY, DISPATCH_DECLARE_CB, __VA_ARGS__) \ + MTARGETS_CONF_BASELINE(DISPATCH_DECLARE_BASE_CB, __VA_ARGS__) + +// Preprocessor callbacks +#define DISPATCH_DECLARE_CB(TESTED_FEATURES_DUMMY, TARGET_NAME, LEFT, ...) \ + CAT(CAT(LEFT, _), TARGET_NAME) __VA_ARGS__; +#define DISPATCH_DECLARE_BASE_CB(LEFT, ...) \ + LEFT __VA_ARGS__; + +#define DISPATCH_CALL(NAME) \ + ( \ + MTARGETS_CONF_DISPATCH(CPU_TEST, DISPATCH_CALL_CB, NAME) \ + MTARGETS_CONF_BASELINE(DISPATCH_CALL_BASE_CB, NAME) \ + NULL \ + ) +// Preprocessor callbacks +#define DISPATCH_CALL_CB(TESTED_FEATURES, TARGET_NAME, LEFT) \ + (TESTED_FEATURES) ? CAT(CAT(LEFT, _), TARGET_NAME) : +#define DISPATCH_CALL_BASE_CB(LEFT) \ + (1) ? LEFT : + +#include "dispatch1.conf.h" +DISPATCH_DECLARE(const char *dispatch1, ()) + +#include "dispatch2.conf.h" +DISPATCH_DECLARE(const char *dispatch2, ()) + +#endif // DISPATCH_H_ diff --git a/test cases/features/2 multi_targets/dispatch1.c b/test cases/features/2 multi_targets/dispatch1.c new file mode 100644 index 000000000000..b4463355e2a5 --- /dev/null +++ b/test cases/features/2 multi_targets/dispatch1.c @@ -0,0 +1,28 @@ +#include "dispatch.h" + +const char *DISPATCH_CURRENT(dispatch1)() +{ +#ifdef HAVE_SSSE3 + #ifndef HAVE_SSE3 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_SSE2 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_SSE + #error "expected a defention for implied features" + #endif +#endif +#ifdef HAVE_ASIMD + #ifndef HAVE_NEON + #error "expected a defention for implied features" + #endif + #ifndef HAVE_NEON_FP16 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_NEON_VFPV4 + #error "expected a defention for implied features" + #endif +#endif + return TOSTRING(DISPATCH_CURRENT(dispatch1)); +} diff --git a/test cases/features/2 multi_targets/dispatch2.c b/test cases/features/2 multi_targets/dispatch2.c new file mode 100644 index 000000000000..7a69fc3d0375 --- /dev/null +++ b/test cases/features/2 multi_targets/dispatch2.c @@ -0,0 +1,21 @@ +#include "dispatch.h" + +const char *DISPATCH_CURRENT(dispatch2)() +{ +#ifdef HAVE_SSE41 + #ifndef HAVE_SSSE3 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_SSE3 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_SSE2 + #error "expected a defention for implied features" + #endif + #ifndef HAVE_SSE + #error "expected a defention for implied features" + #endif +#endif + return TOSTRING(DISPATCH_CURRENT(dispatch2)); +} + diff --git a/test cases/features/2 multi_targets/dispatch3.c b/test cases/features/2 multi_targets/dispatch3.c new file mode 100644 index 000000000000..a3a555af2ca7 --- /dev/null +++ b/test cases/features/2 multi_targets/dispatch3.c @@ -0,0 +1 @@ +#error "no targets enabled for this source" diff --git a/test cases/features/2 multi_targets/init_features b/test cases/features/2 multi_targets/init_features new file mode 120000 index 000000000000..d52da150db76 --- /dev/null +++ b/test cases/features/2 multi_targets/init_features @@ -0,0 +1 @@ +../init_features \ No newline at end of file diff --git a/test cases/features/2 multi_targets/main.c b/test cases/features/2 multi_targets/main.c new file mode 100644 index 000000000000..88ea11ae767c --- /dev/null +++ b/test cases/features/2 multi_targets/main.c @@ -0,0 +1,45 @@ +#include "dispatch.h" +#include +#include + +int cpu_has(int feature_id) +{ + // we assume the used features are supported by CPU + return 1; +} + +int main() +{ + #include "dispatch1.conf.h" + const char *dispatch1_str = DISPATCH_CALL(dispatch1)(); +#if defined(TEST_X86) + const char *exp_dispatch1_str = "dispatch1_SSSE3"; +#elif defined(TEST_ARM64) + const char *exp_dispatch1_str = "dispatch1"; +#elif defined(TEST_ARM) + const char *exp_dispatch1_str = "dispatch1_ASIMD"; +#else + const char *exp_dispatch1_str = "dispatch1"; +#endif + if (strcmp(dispatch1_str, exp_dispatch1_str) != 0) { + return 1; + } + #include "dispatch2.conf.h" + const char *dispatch2_str = DISPATCH_CALL(dispatch2)(); +#if defined(TEST_X86) + const char *exp_dispatch2_str = "dispatch2_SSE41"; +#elif defined(TEST_ARM64) || defined(TEST_ARM) + const char *exp_dispatch2_str = "dispatch2_ASIMD"; +#else + const char *exp_dispatch2_str = "dispatch2"; +#endif + if (strcmp(dispatch2_str, exp_dispatch2_str) != 0) { + return 2; + } + #include "dispatch3.conf.h" + const char *dispatch3_str = DISPATCH_CALL(dispatch3); + if (dispatch3_str != NULL) { + return 3; + } + return 0; +} diff --git a/test cases/features/2 multi_targets/meson.build b/test cases/features/2 multi_targets/meson.build new file mode 100644 index 000000000000..ffceda098f1a --- /dev/null +++ b/test cases/features/2 multi_targets/meson.build @@ -0,0 +1,26 @@ +project('multi_targets', 'c') +subdir('init_features') + +multi_targets = mod_features.multi_targets( + 'dispatch1.conf.h', 'dispatch1.c', + dispatch: [SSSE3, ASIMD], + baseline: [SSE3, NEON] +) + +multi_targets.extend(mod_features.multi_targets( + 'dispatch2.conf.h', 'dispatch2.c', + dispatch: [SSE41, SSSE3, ASIMD], + baseline: [SSE3] +)) + +multi_targets.extend(mod_features.multi_targets( + 'dispatch3.conf.h', 'dispatch3.c', + dispatch: [] +)) + +exe = executable( + 'multi_targets', 'main.c', + link_with: multi_targets.static_lib('multi_targets_lib') +) + +test('multi_targets', exe) diff --git a/test cases/features/init_features/checks/cpu_asimd.c b/test cases/features/init_features/checks/cpu_asimd.c new file mode 100644 index 000000000000..6bc9022a58d3 --- /dev/null +++ b/test cases/features/init_features/checks/cpu_asimd.c @@ -0,0 +1,27 @@ +#ifdef _MSC_VER + #include +#endif +#include + +int main(int argc, char **argv) +{ + float *src = (float*)argv[argc-1]; + float32x4_t v1 = vdupq_n_f32(src[0]), v2 = vdupq_n_f32(src[1]); + /* MAXMIN */ + int ret = (int)vgetq_lane_f32(vmaxnmq_f32(v1, v2), 0); + ret += (int)vgetq_lane_f32(vminnmq_f32(v1, v2), 0); + /* ROUNDING */ + ret += (int)vgetq_lane_f32(vrndq_f32(v1), 0); +#ifdef __aarch64__ + { + double *src2 = (double*)argv[argc-1]; + float64x2_t vd1 = vdupq_n_f64(src2[0]), vd2 = vdupq_n_f64(src2[1]); + /* MAXMIN */ + ret += (int)vgetq_lane_f64(vmaxnmq_f64(vd1, vd2), 0); + ret += (int)vgetq_lane_f64(vminnmq_f64(vd1, vd2), 0); + /* ROUNDING */ + ret += (int)vgetq_lane_f64(vrndq_f64(vd1), 0); + } +#endif + return ret; +} diff --git a/test cases/features/init_features/checks/cpu_neon.c b/test cases/features/init_features/checks/cpu_neon.c new file mode 100644 index 000000000000..8c64f864dea6 --- /dev/null +++ b/test cases/features/init_features/checks/cpu_neon.c @@ -0,0 +1,19 @@ +#ifdef _MSC_VER + #include +#endif +#include + +int main(int argc, char **argv) +{ + // passing from untraced pointers to avoid optimizing out any constants + // so we can test against the linker. + float *src = (float*)argv[argc-1]; + float32x4_t v1 = vdupq_n_f32(src[0]), v2 = vdupq_n_f32(src[1]); + int ret = (int)vgetq_lane_f32(vmulq_f32(v1, v2), 0); +#ifdef __aarch64__ + double *src2 = (double*)argv[argc-2]; + float64x2_t vd1 = vdupq_n_f64(src2[0]), vd2 = vdupq_n_f64(src2[1]); + ret += (int)vgetq_lane_f64(vmulq_f64(vd1, vd2), 0); +#endif + return ret; +} diff --git a/test cases/features/init_features/checks/cpu_neon_fp16.c b/test cases/features/init_features/checks/cpu_neon_fp16.c new file mode 100644 index 000000000000..f3b949770db6 --- /dev/null +++ b/test cases/features/init_features/checks/cpu_neon_fp16.c @@ -0,0 +1,11 @@ +#ifdef _MSC_VER + #include +#endif +#include + +int main(int argc, char **argv) +{ + short *src = (short*)argv[argc-1]; + float32x4_t v_z4 = vcvt_f32_f16((float16x4_t)vld1_s16(src)); + return (int)vgetq_lane_f32(v_z4, 0); +} diff --git a/test cases/features/init_features/checks/cpu_neon_vfpv4.c b/test cases/features/init_features/checks/cpu_neon_vfpv4.c new file mode 100644 index 000000000000..a039159ddeed --- /dev/null +++ b/test cases/features/init_features/checks/cpu_neon_vfpv4.c @@ -0,0 +1,21 @@ +#ifdef _MSC_VER + #include +#endif +#include + +int main(int argc, char **argv) +{ + float *src = (float*)argv[argc-1]; + float32x4_t v1 = vdupq_n_f32(src[0]); + float32x4_t v2 = vdupq_n_f32(src[1]); + float32x4_t v3 = vdupq_n_f32(src[2]); + int ret = (int)vgetq_lane_f32(vfmaq_f32(v1, v2, v3), 0); +#ifdef __aarch64__ + double *src2 = (double*)argv[argc-2]; + float64x2_t vd1 = vdupq_n_f64(src2[0]); + float64x2_t vd2 = vdupq_n_f64(src2[1]); + float64x2_t vd3 = vdupq_n_f64(src2[2]); + ret += (int)vgetq_lane_f64(vfmaq_f64(vd1, vd2, vd3), 0); +#endif + return ret; +} diff --git a/test cases/features/init_features/checks/cpu_sse.c b/test cases/features/init_features/checks/cpu_sse.c new file mode 100644 index 000000000000..bb98bf63c0b9 --- /dev/null +++ b/test cases/features/init_features/checks/cpu_sse.c @@ -0,0 +1,7 @@ +#include + +int main(void) +{ + __m128 a = _mm_add_ps(_mm_setzero_ps(), _mm_setzero_ps()); + return (int)_mm_cvtss_f32(a); +} diff --git a/test cases/features/init_features/checks/cpu_sse2.c b/test cases/features/init_features/checks/cpu_sse2.c new file mode 100644 index 000000000000..658afc9b4abf --- /dev/null +++ b/test cases/features/init_features/checks/cpu_sse2.c @@ -0,0 +1,7 @@ +#include + +int main(void) +{ + __m128i a = _mm_add_epi16(_mm_setzero_si128(), _mm_setzero_si128()); + return _mm_cvtsi128_si32(a); +} diff --git a/test cases/features/init_features/checks/cpu_sse3.c b/test cases/features/init_features/checks/cpu_sse3.c new file mode 100644 index 000000000000..aece1e60174c --- /dev/null +++ b/test cases/features/init_features/checks/cpu_sse3.c @@ -0,0 +1,7 @@ +#include + +int main(void) +{ + __m128 a = _mm_hadd_ps(_mm_setzero_ps(), _mm_setzero_ps()); + return (int)_mm_cvtss_f32(a); +} diff --git a/test cases/features/init_features/checks/cpu_sse41.c b/test cases/features/init_features/checks/cpu_sse41.c new file mode 100644 index 000000000000..bfdb9feacc47 --- /dev/null +++ b/test cases/features/init_features/checks/cpu_sse41.c @@ -0,0 +1,7 @@ +#include + +int main(void) +{ + __m128 a = _mm_floor_ps(_mm_setzero_ps()); + return (int)_mm_cvtss_f32(a); +} diff --git a/test cases/features/init_features/checks/cpu_ssse3.c b/test cases/features/init_features/checks/cpu_ssse3.c new file mode 100644 index 000000000000..ad0abc1e66fb --- /dev/null +++ b/test cases/features/init_features/checks/cpu_ssse3.c @@ -0,0 +1,7 @@ +#include + +int main(void) +{ + __m128i a = _mm_hadd_epi16(_mm_setzero_si128(), _mm_setzero_si128()); + return (int)_mm_cvtsi128_si32(a); +} diff --git a/test cases/features/init_features/meson.build b/test cases/features/init_features/meson.build new file mode 100644 index 000000000000..c0b4f13fa862 --- /dev/null +++ b/test cases/features/init_features/meson.build @@ -0,0 +1,98 @@ +#project('test-features', 'c') +mod_features = import('features') +cpu_family = host_machine.cpu_family() +compiler_id = meson.get_compiler('c').get_id() +source_root = meson.project_source_root() + '/../init_features/' +# Basic X86 Features +# ------------------ +SSE = mod_features.new( + 'SSE', 1, args: '-msse', + test_code: files(source_root + 'checks/cpu_sse.c')[0] +) +SSE2 = mod_features.new( + 'SSE2', 2, implies: SSE, + args: '-msse2', + test_code: files(source_root + 'checks/cpu_sse2.c')[0] +) +# enabling SSE without SSE2 is useless also +# it's non-optional for x86_64 +SSE.update(implies: SSE2) +SSE3 = mod_features.new( + 'SSE3', 3, implies: SSE2, + args: '-msse3', + test_code: files(source_root + 'checks/cpu_sse3.c')[0] +) +SSSE3 = mod_features.new( + 'SSSE3', 4, implies: SSE3, + args: '-mssse3', + test_code: files(source_root + 'checks/cpu_ssse3.c')[0] +) +SSE41 = mod_features.new( + 'SSE41', 5, implies: SSSE3, + args: '-msse4.1', + test_code: files(source_root + 'checks/cpu_sse41.c')[0] +) +if cpu_family not in ['x86', 'x86_64'] + # should disable any prevalent features + SSE.update(disable: 'not supported by the current platform') +endif +# Specializations for non unix-like compilers +if compiler_id == 'intel-cl' + foreach fet : [SSE, SSE2, SSE3, SSSE3] + fet.update(args: {'val': '/arch:' + fet.get('name'), 'match': '/arch:.*'}) + endforeach + SSE41.update(args: {'val': '/arch:SSE4.1', 'match': '/arch:.*'}) +elif compiler_id == 'msvc' + # only available on 32-bit. Its enabled by default on 64-bit mode + foreach fet : [SSE, SSE2] + if cpu_family == 'x86' + fet.update(args: {'val': '/arch:' + fet.get('name'), 'match': clear_arch}) + else + fet.update(args: '') + endif + endforeach + # The following features don't own private FLAGS still + # the compiler provides ISA capability for them. + foreach fet : [SSE3, SSSE3, SSE41] + fet.update(args: '') + endforeach +endif + +# Basic ARM Features +# ------------------ +NEON = mod_features.new( + 'NEON', 200, + test_code: files(source_root + 'checks/cpu_neon.c')[0] +) +NEON_FP16 = mod_features.new( + 'NEON_FP16', 201, implies: NEON, + test_code: files(source_root + 'checks/cpu_neon_fp16.c')[0] +) +# FMA +NEON_VFPV4 = mod_features.new( + 'NEON_VFPV4', 202, implies: NEON_FP16, + test_code: files(source_root + 'checks/cpu_neon_vfpv4.c')[0] +) +# Advanced SIMD +ASIMD = mod_features.new( + 'ASIMD', 203, implies: NEON_VFPV4, detect: {'val': 'ASIMD', 'match': 'NEON.*'}, + test_code: files(source_root + 'checks/cpu_asimd.c')[0] +) +if cpu_family == 'aarch64' + # hardware baseline, they can't be enabled independently + NEON.update(implies: [NEON_FP16, NEON_VFPV4, ASIMD]) + NEON_FP16.update(implies: [NEON, NEON_VFPV4, ASIMD]) + NEON_VFPV4.update(implies: [NEON, NEON_FP16, ASIMD]) +elif cpu_family == 'arm' + NEON.update(args: '-mfpu=neon') + NEON_FP16.update(args: ['-mfp16-format=ieee', {'val': '-mfpu=neon-fp16', 'match': '-mfpu=.*'}]) + NEON_VFPV4.update(args: {'val': '-mfpu=neon-vfpv4', 'match': '-mfpu=.*'}) + ASIMD.update(args: [ + {'val': '-mfpu=neon-fp-armv8', 'match': '-mfpu=.*'}, + '-march=armv8-a+simd' + ]) +else + # should disable any prevalent features + NEON.update(disable: 'not supported by the current platform') +endif + diff --git a/unittests/featurestests.py b/unittests/featurestests.py new file mode 100644 index 000000000000..a863a9fb93cc --- /dev/null +++ b/unittests/featurestests.py @@ -0,0 +1,300 @@ +# Copyright (c) 2023, NumPy Developers. + +import re +import contextlib +from mesonbuild.interpreter import Interpreter +from mesonbuild.build import Build +from mesonbuild.mparser import FunctionNode, ArgumentNode, Token +from mesonbuild.modules import ModuleState +from mesonbuild.modules.features import Module +from mesonbuild.compilers import Compiler, CompileResult +from mesonbuild.mesonlib import MachineChoice +from mesonbuild.envconfig import MachineInfo + +from .baseplatformtests import BasePlatformTests +from run_tests import get_convincing_fake_env_and_cc + +class FakeCompiler(Compiler): + language = 'c' + + def __init__(self, trap_args = '', trap_code=''): + super().__init__( + ccache=[], exelist=[], version='0.0', + for_machine=MachineChoice.HOST, + info=MachineInfo( + system='linux', cpu_family='x86_64', + cpu='xeon', endian='little', + kernel='linux', subsystem='numpy' + ), + is_cross=True + ) + self.trap_args = trap_args + self.trap_code = trap_code + + def sanity_check(self, work_dir: str, environment: 'Environment') -> None: + pass + + def get_optimization_args(self, optimization_level: str) -> 'T.List[str]': + return [] + + def get_output_args(self, outputname: str) -> 'T.List[str]': + return [] + + def has_multi_arguments(self, args: 'T.List[str]', env: 'Environment') -> 'T.Tuple[bool, bool]': + if self.trap_args: + for a in args: + if re.match(self.trap_args, a): + return False, False + return True, False + + @contextlib.contextmanager + def compile(self, code: 'mesonlib.FileOrString', *args, **kwargs + ) -> 'T.Iterator[T.Optional[CompileResult]]': + if self.trap_code and re.match(self.trap_code, code): + rcode = -1 + else: + rcode = 0 + result = CompileResult(returncode=rcode) + yield result + + @contextlib.contextmanager + def cached_compile(self, code: 'mesonlib.FileOrString', *args, **kwargs + ) -> 'T.Iterator[T.Optional[CompileResult]]': + if self.trap_code and re.match(self.trap_code, code): + rcode = -1 + else: + rcode = 0 + result = CompileResult(returncode=rcode) + yield result + +class FeaturesTests(BasePlatformTests): + def setUp(self): + super().setUp() + env, cc = get_convincing_fake_env_and_cc( + bdir=self.builddir, prefix=self.prefix) + env.machines.target = env.machines.host + + build = Build(env) + interp = Interpreter(build, mock=True) + project = interp.funcs['project'] + filename = 'featurestests.py' + node = FunctionNode( + filename = filename, + lineno = 0, + colno = 0, + end_lineno = 0, + end_colno = 0, + func_name = 'FeaturesTests', + args = ArgumentNode(Token('string', filename, 0, 0, 0, None, '')) + ) + project(node, ['Test Module Features'], {'version': '0.1'}) + self.cc = cc + self.state = ModuleState(interp) + self.mod_features = Module() + + def clear_cache(self): + self.mod_features = Module() + + def mod_method(self, name: str, *args, **kwargs): + mth = self.mod_features.methods.get(name) + return mth(self.state, list(args), kwargs) + + def mod_new(self, *args, **kwargs): + return self.mod_method('new', *args, **kwargs) + + def mod_test(self, *args, **kwargs): + return self.mod_method('test', *args, **kwargs) + + def update_feature(self, feature, **kwargs): + feature.update_method(self.state, [], kwargs) + + def check_result(self, features, expected_result, anyfet=False, **kwargs): + is_supported, test_result = self.mod_test( + *features, compiler=FakeCompiler(**kwargs), + cached=False, anyfet=anyfet + ) + test_result = test_result.copy() # to avoid pop cached dict + test_result.pop('fail_reason') + self.assertEqual(is_supported, expected_result['is_supported']) + self.assertEqual(test_result, expected_result) + + def gen_basic_result(self, prevalent, predecessor=[]): + prevalent_names = [fet.name for fet in prevalent] + predecessor_names = [fet.name for fet in predecessor] + features_names = predecessor_names + prevalent_names + return { + 'target_name': '__'.join(prevalent_names), + 'prevalent_features': prevalent_names, + 'features': features_names, + 'args': [f'arg{i}' for i in range(1, len(features_names) + 1)], + 'detect': features_names, + 'defines': features_names, + 'undefines': [], + 'is_supported': True, + 'is_disabled': False + } + + def gen_fail_result(self, prevalent, is_supported=False, + is_disabled = False): + prevalent_names = [fet.name for fet in prevalent] + return { + 'target_name': '__'.join(prevalent_names), + 'prevalent_features': prevalent_names, + 'features': [], + 'args': [], + 'detect': [], + 'defines': [], + 'undefines': [], + 'is_supported': is_supported, + 'is_disabled': is_disabled + } + + def test_happy_path(self): + fet1 = self.mod_new('fet1', 1, args='arg1', test_code='test1') + fet2 = self.mod_new('fet2', 2, implies=fet1, args='arg2', test_code='test2') + fet3 = self.mod_new('fet3', 3, implies=fet2, args='arg3') + fet4 = self.mod_new('fet4', 4, implies=fet3, args='arg4', test_code='test4') + # fet5 doesn't imply fet4 so we can test target with muti prevalent features + fet5 = self.mod_new('fet5', 5, implies=fet3, args='arg5') + fet6 = self.mod_new('fet6', 6, implies=[fet4, fet5], args='arg6') + + # basic test expected the compiler support all operations + for test_features, prevalent, predecessor in [ + ([fet1], [fet1], []), + ([fet2], [fet2], [fet1]), + ([fet2, fet1], [fet2], [fet1]), + ([fet3], [fet3], [fet1, fet2]), + ([fet2, fet3, fet1], [fet3], [fet1, fet2]), + ([fet4, fet5], [fet4, fet5], [fet1, fet2, fet3]), + ([fet5, fet4], [fet4, fet5], [fet1, fet2, fet3]), + ([fet6], [fet6], [fet1, fet2, fet3, fet4, fet5]), + ]: + expected = self.gen_basic_result(prevalent, predecessor) + self.check_result(test_features, expected) + + for test_features, prevalent, trap_args in [ + ([fet1], [fet1], 'arg1'), + ([fet1], [fet1], 'arg1'), + ]: + expected = self.gen_fail_result(prevalent) + self.check_result(test_features, expected, trap_args=trap_args) + + def test_failures(self): + fet1 = self.mod_new('fet1', 1, args='arg1', test_code='test1') + fet2 = self.mod_new('fet2', 2, implies=fet1, args='arg2', test_code='test2') + fet3 = self.mod_new('fet3', 3, implies=fet2, args='arg3', test_code='test3') + fet4 = self.mod_new('fet4', 4, implies=fet3, args='arg4', test_code='test4') + # fet5 doesn't imply fet4 so we can test target with muti features + fet5 = self.mod_new('fet5', 5, implies=fet3, args='arg5', test_code='test5') + + for test_features, prevalent, disable, trap_args, trap_code in [ + # test by trap flags + ([fet1], [fet1], None, 'arg1', None), + ([fet2], [fet2], None, 'arg1', None), + ([fet2, fet1], [fet2], None, 'arg2', None), + ([fet3], [fet3], None, 'arg1', None), + ([fet3, fet2], [fet3], None, 'arg2', None), + ([fet3, fet1], [fet3], None, 'arg3', None), + ([fet3, fet1], [fet3], None, 'arg3', None), + ([fet5, fet4], [fet4, fet5], None, 'arg4', None), + ([fet5, fet4], [fet4, fet5], None, 'arg5', None), + # test by trap test_code + ([fet1], [fet1], None, None, 'test1'), + ([fet2], [fet2], None, None, 'test1'), + ([fet2, fet1], [fet2], None, None, 'test2'), + ([fet3], [fet3], None, None, 'test1'), + ([fet3, fet2], [fet3], None, None, 'test2'), + ([fet3, fet1], [fet3], None, None, 'test3'), + ([fet5, fet4], [fet4, fet5], None, None, 'test4'), + ([fet5, fet4], [fet4, fet5], None, None, 'test5'), + # test by disable feature + ([fet1], [fet1], fet1, None, None), + ([fet2], [fet2], fet1, None, None), + ([fet2, fet1], [fet2], fet2, None, None), + ([fet3], [fet3], fet1, None, None), + ([fet3, fet2], [fet3], fet2, None, None), + ([fet3, fet1], [fet3], fet3, None, None), + ([fet5, fet4], [fet4, fet5], fet4, None, None), + ([fet5, fet4], [fet4, fet5], fet5, None, None), + ]: + if disable: + self.update_feature(disable, disable='test disable') + expected = self.gen_fail_result(prevalent, is_disabled=not not disable) + self.check_result(test_features, expected, trap_args=trap_args, trap_code=trap_code) + + def test_any(self): + fet1 = self.mod_new('fet1', 1, args='arg1', test_code='test1') + fet2 = self.mod_new('fet2', 2, implies=fet1, args='arg2', test_code='test2') + fet3 = self.mod_new('fet3', 3, implies=fet2, args='arg3', test_code='test3') + fet4 = self.mod_new('fet4', 4, implies=fet3, args='arg4', test_code='test4') + # fet5 doesn't imply fet4 so we can test target with muti features + fet5 = self.mod_new('fet5', 5, implies=fet3, args='arg5', test_code='test5') + fet6 = self.mod_new('fet6', 6, implies=[fet4, fet5], args='arg6') + + for test_features, prevalent, predecessor, trap_args in [ + ([fet2], [fet1], [], 'arg2'), + ([fet6], [fet2], [fet1], 'arg3'), + ([fet6], [fet4], [fet1, fet2, fet3], 'arg5'), + ([fet5, fet4], [fet3], [fet1, fet2], 'arg4|arg5'), + ]: + expected = self.gen_basic_result(prevalent, predecessor) + self.check_result(test_features, expected, trap_args=trap_args, anyfet=True) + + def test_conflict_args(self): + fet1 = self.mod_new('fet1', 1, args='arg1', test_code='test1') + fet2 = self.mod_new('fet2', 2, implies=fet1, args='arg2', test_code='test2') + fet3 = self.mod_new('fet3', 3, implies=fet2, args='arg3') + fet4 = self.mod_new('fet4', 4, implies=fet3, args='arg4', test_code='test4') + fet5 = self.mod_new('fet5', 5, implies=fet3, args='arg5', test_code='test5') + fet6 = self.mod_new('fet6', 6, implies=[fet4, fet5], args='arch=xx') + + compiler = FakeCompiler() + for implies, attr, val, expected_vals in [ + ( + [fet3, fet4, fet5], + 'args', {'val':'arch=the_arch', 'match': 'arg.*'}, + ['arch=the_arch'], + ), + ( + [fet5, fet4], + 'args', {'val':'arch=', 'match': 'arg.*', 'mfilter': '[0-9]'}, + ['arch=12345'], + ), + ( + [fet5, fet4], + 'args', {'val':'arch=', 'match': 'arg.*', 'mfilter': '[0-9]', 'mjoin': '+'}, + ['arch=1+2+3+4+5'], + ), + ( + [fet5, fet4], + 'args', {'val':'arch=num*', 'match': 'arg.*[0-3]', 'mfilter': '[0-9]', 'mjoin': '*'}, + ['arg4', 'arg5', 'arch=num*1*2*3'], + ), + ( + [fet6], + 'args', {'val':'arch=', 'match': 'arg.*[0-9]|arch=.*', 'mfilter': '([0-9])|arch=(\w+)', 'mjoin': '*'}, + ['arch=1*2*3*4*5*xx'], + ), + ( + [fet3, fet4, fet5], + 'detect', {'val':'test_fet', 'match': 'fet.*[0-5]'}, + ['test_fet'], + ), + ( + [fet5, fet4], + 'detect', {'val':'fet', 'match': 'fet.*[0-5]', 'mfilter': '[0-9]'}, + ['fet12345'], + ), + ( + [fet5, fet4], + 'detect', {'val':'fet_', 'match': 'fet.*[0-5]', 'mfilter': '[0-9]', 'mjoin':'_'}, + ['fet_1_2_3_4_5'], + ), + ]: + test_fet = self.mod_new('test_fet', 7, implies=implies, **{attr:val}) + is_supported, test_result = self.mod_test( + test_fet, compiler=compiler, cached=False + ) + self.assertEqual(test_result['is_supported'], True) + self.assertEqual(test_result[attr], expected_vals) +