Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to include built-in include files #94193

Merged
merged 1 commit into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions doc/classes/ShaderIncludeDB.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<?xml version="1.0" encoding="UTF-8" ?>
<class name="ShaderIncludeDB" inherits="Object" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../class.xsd">
<brief_description>
Internal database of built in shader include files.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Internal database of built in shader include files.
Internal database of built-in shader include files.

</brief_description>
<description>
This object contains shader fragments from Godot's internal shaders. These can be used when access to internal uniform buffers and/or internal functions is required for instance when composing compositor effects or compute shaders. Only fragments for the current rendering device are loaded.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This object contains shader fragments from Godot's internal shaders. These can be used when access to internal uniform buffers and/or internal functions is required for instance when composing compositor effects or compute shaders. Only fragments for the current rendering device are loaded.
This object contains shader fragments from Godot's internal shaders. These can be used when it is necessary to access to internal uniform buffers and/or internal functions, for example when composing compositor effects or compute shaders. Only fragments for the current rendering device are loaded.

</description>
<tutorials>
</tutorials>
<methods>
<method name="get_built_in_include_file" qualifiers="static">
<return type="String" />
<param index="0" name="filename" type="String" />
<description>
Returns the code for the built-in shader fragment. You can also access this in your shader code through [code]#include "filename"[/code].
</description>
</method>
<method name="has_built_in_include_file" qualifiers="static">
<return type="bool" />
<param index="0" name="filename" type="String" />
<description>
Returns [code]true[/code] if an include file with this name exists.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Returns [code]true[/code] if an include file with this name exists.
Returns [code]true[/code] if an include file with the given [param filename] exists.

</description>
</method>
<method name="list_built_in_include_files" qualifiers="static">
<return type="PackedStringArray" />
<description>
Returns a list of built-in include files that are currently registered.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Returns a list of built-in include files that are currently registered.
Returns the list of built-in include files that are currently registered.

</description>
</method>
</methods>
</class>
2 changes: 2 additions & 0 deletions servers/register_server_types.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
#include "rendering/renderer_rd/uniform_set_cache_rd.h"
#include "rendering/rendering_device.h"
#include "rendering/rendering_device_binds.h"
#include "rendering/shader_include_db.h"
#include "rendering/storage/render_data.h"
#include "rendering/storage/render_scene_buffers.h"
#include "rendering/storage/render_scene_data.h"
Expand Down Expand Up @@ -210,6 +211,7 @@ void register_server_types() {
}

GDREGISTER_ABSTRACT_CLASS(RenderingDevice);
GDREGISTER_CLASS(ShaderIncludeDB);
GDREGISTER_CLASS(RDTextureFormat);
GDREGISTER_CLASS(RDTextureView);
GDREGISTER_CLASS(RDAttachmentFormat);
Expand Down
11 changes: 11 additions & 0 deletions servers/rendering/renderer_rd/renderer_scene_render_rd.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,13 @@
#include "core/os/os.h"
#include "renderer_compositor_rd.h"
#include "servers/rendering/renderer_rd/environment/fog.h"
#include "servers/rendering/renderer_rd/shaders/decal_data_inc.glsl.gen.h"
#include "servers/rendering/renderer_rd/shaders/light_data_inc.glsl.gen.h"
#include "servers/rendering/renderer_rd/shaders/scene_data_inc.glsl.gen.h"
#include "servers/rendering/renderer_rd/storage_rd/material_storage.h"
#include "servers/rendering/renderer_rd/storage_rd/texture_storage.h"
#include "servers/rendering/rendering_server_default.h"
#include "servers/rendering/shader_include_db.h"
#include "servers/rendering/storage/camera_attributes_storage.h"

void get_vogel_disk(float *r_kernel, int p_sample_count) {
Expand Down Expand Up @@ -1452,6 +1456,13 @@ void RendererSceneRenderRD::init() {
/* Forward ID */
forward_id_storage = create_forward_id_storage();

/* Register the include files we make available by default to our users */
{
ShaderIncludeDB::register_built_in_include_file("godot/decal_data_inc.glsl", decal_data_inc_shader_glsl);
ShaderIncludeDB::register_built_in_include_file("godot/light_data_inc.glsl", light_data_inc_shader_glsl);
ShaderIncludeDB::register_built_in_include_file("godot/scene_data_inc.glsl", scene_data_inc_shader_glsl);
}

/* SKY SHADER */

sky.init();
Expand Down
33 changes: 32 additions & 1 deletion servers/rendering/renderer_rd/shader_rd.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
#include "core/version.h"
#include "renderer_compositor_rd.h"
#include "servers/rendering/rendering_device.h"
#include "servers/rendering/shader_include_db.h"
#include "thirdparty/misc/smolv.h"

#define ENABLE_SHADER_CACHE 1
Expand All @@ -46,7 +47,8 @@ void ShaderRD::_add_stage(const char *p_code, StageType p_stage_type) {

String text;

for (int i = 0; i < lines.size(); i++) {
int line_count = lines.size();
for (int i = 0; i < line_count; i++) {
const String &l = lines[i];
bool push_chunk = false;

Expand Down Expand Up @@ -78,6 +80,35 @@ void ShaderRD::_add_stage(const char *p_code, StageType p_stage_type) {
chunk.type = StageTemplate::Chunk::TYPE_CODE;
push_chunk = true;
chunk.code = l.replace_first("#CODE", String()).replace(":", "").strip_edges().to_upper();
} else if (l.begins_with("#include ")) {
String include_file = l.replace("#include ", "").strip_edges();
if (include_file[0] == '"') {
int end_pos = include_file.find_char('"', 1);
if (end_pos >= 0) {
include_file = include_file.substr(1, end_pos - 1);

String include_code = ShaderIncludeDB::get_built_in_include_file(include_file);
if (!include_code.is_empty()) {
// Add these lines into our parse list so we parse them as well.
Vector<String> include_lines = include_code.split("\n");

for (int j = include_lines.size() - 1; j >= 0; j--) {
lines.insert(i + 1, include_lines[j]);
}

line_count = lines.size();
} else {
// Add it in as is.
text += l + "\n";
}
} else {
// Add it in as is.
text += l + "\n";
}
} else {
// Add it in as is.
text += l + "\n";
}
} else {
text += l + "\n";
}
Expand Down
10 changes: 7 additions & 3 deletions servers/rendering/renderer_rd/shaders/SCsub
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,20 @@ from misc.utility.scons_hints import *
Import("env")

if "RD_GLSL" in env["BUILDERS"]:
# find all include files
# find just the include files
gl_include_files = [str(f) for f in Glob("*_inc.glsl")]

# find all shader code(all glsl files excluding our include files)
# find all shader code (all glsl files excluding our include files)
glsl_files = [str(f) for f in Glob("*.glsl") if str(f) not in gl_include_files]

# make sure we recompile shaders if include files change
env.Depends([f + ".gen.h" for f in glsl_files], gl_include_files + ["#glsl_builders.py"])

# compile shaders
# compile include files
for glsl_file in gl_include_files:
env.GLSL_HEADER(glsl_file)

# compile RD shader
for glsl_file in glsl_files:
env.RD_GLSL(glsl_file)

Expand Down
7 changes: 6 additions & 1 deletion servers/rendering/rendering_device.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
#include "rendering_device.compat.inc"

#include "rendering_device_binds.h"
#include "shader_include_db.h"

#include "core/config/project_settings.h"
#include "core/io/dir_access.h"
Expand Down Expand Up @@ -189,6 +190,10 @@ void RenderingDevice::_free_dependencies(RID p_id) {
}
}

/*******************************/
/**** SHADER INFRASTRUCTURE ****/
/*******************************/

void RenderingDevice::shader_set_compile_to_spirv_function(ShaderCompileToSPIRVFunction p_function) {
compile_to_spirv_function = p_function;
}
Expand All @@ -211,7 +216,7 @@ Vector<uint8_t> RenderingDevice::shader_compile_spirv_from_source(ShaderStage p_

ERR_FAIL_NULL_V(compile_to_spirv_function, Vector<uint8_t>());

return compile_to_spirv_function(p_stage, p_source_code, p_language, r_error, this);
return compile_to_spirv_function(p_stage, ShaderIncludeDB::parse_include_files(p_source_code), p_language, r_error, this);
}

String RenderingDevice::shader_get_spirv_cache_key() const {
Expand Down
16 changes: 12 additions & 4 deletions servers/rendering/rendering_device_binds.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@

#include "rendering_device_binds.h"

#include "shader_include_db.h"
BastiaanOlij marked this conversation as resolved.
Show resolved Hide resolved

Error RDShaderFile::parse_versions_from_text(const String &p_text, const String p_defines, OpenIncludeFunction p_include_func, void *p_include_func_userdata) {
ERR_FAIL_NULL_V_MSG(
RenderingDevice::get_singleton(),
Expand Down Expand Up @@ -144,11 +146,17 @@ Error RDShaderFile::parse_versions_from_text(const String &p_text, const String
break;
}
include = include.substr(1, include.length() - 2).strip_edges();
String include_text = p_include_func(include, p_include_func_userdata);
if (!include_text.is_empty()) {
stage_code[stage] += "\n" + include_text + "\n";

String include_code = ShaderIncludeDB::get_built_in_include_file(include);
if (!include_code.is_empty()) {
stage_code[stage] += "\n" + include_code + "\n";
} else {
base_error = "#include failed for file '" + include + "'";
String include_text = p_include_func(include, p_include_func_userdata);
if (!include_text.is_empty()) {
stage_code[stage] += "\n" + include_text + "\n";
} else {
base_error = "#include failed for file '" + include + "'.";
}
}
} else {
base_error = "#include used, but no include function provided.";
Expand Down
115 changes: 115 additions & 0 deletions servers/rendering/shader_include_db.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
/**************************************************************************/
/* shader_include_db.cpp */
/**************************************************************************/
/* This file is part of: */
/* GODOT ENGINE */
/* https://godotengine.org */
/**************************************************************************/
/* Copyright (c) 2014-present Godot Engine contributors (see AUTHORS.md). */
/* Copyright (c) 2007-2014 Juan Linietsky, Ariel Manzur. */
/* */
/* Permission is hereby granted, free of charge, to any person obtaining */
/* a copy of this software and associated documentation files (the */
/* "Software"), to deal in the Software without restriction, including */
/* without limitation the rights to use, copy, modify, merge, publish, */
/* distribute, sublicense, and/or sell copies of the Software, and to */
/* permit persons to whom the Software is furnished to do so, subject to */
/* the following conditions: */
/* */
/* The above copyright notice and this permission notice shall be */
/* included in all copies or substantial portions of the Software. */
/* */
/* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, */
/* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF */
/* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. */
/* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY */
/* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, */
/* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE */
/* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
/**************************************************************************/

#include "shader_include_db.h"

HashMap<String, String> ShaderIncludeDB::built_in_includes;

void ShaderIncludeDB::_bind_methods() {
ClassDB::bind_static_method("ShaderIncludeDB", D_METHOD("list_built_in_include_files"), &ShaderIncludeDB::list_built_in_include_files);
ClassDB::bind_static_method("ShaderIncludeDB", D_METHOD("has_built_in_include_file", "filename"), &ShaderIncludeDB::has_built_in_include_file);
ClassDB::bind_static_method("ShaderIncludeDB", D_METHOD("get_built_in_include_file", "filename"), &ShaderIncludeDB::get_built_in_include_file);
}

void ShaderIncludeDB::register_built_in_include_file(const String &p_filename, const String &p_shader_code) {
built_in_includes[p_filename] = p_shader_code;
}

PackedStringArray ShaderIncludeDB::list_built_in_include_files() {
PackedStringArray ret;

for (const KeyValue<String, String> &e : built_in_includes) {
ret.push_back(e.key);
}

return ret;
}

bool ShaderIncludeDB::has_built_in_include_file(const String &p_filename) {
return built_in_includes.has(p_filename);
}

String ShaderIncludeDB::get_built_in_include_file(const String &p_filename) {
const String *ptr = built_in_includes.getptr(p_filename);

return ptr ? *ptr : String();
}

String ShaderIncludeDB::parse_include_files(const String &p_code) {
// Prevent needless processing if we don't have any includes.
if (p_code.find("#include ") == -1) {
return p_code;
}

const String include = "#include ";
String parsed_code;

Vector<String> lines = p_code.split("\n");
int line_count = lines.size();
for (int i = 0; i < line_count; i++) {
const String &l = lines[i];

if (l.begins_with(include)) {
String include_file = l.replace(include, "").strip_edges();
if (include_file[0] == '"') {
int end_pos = include_file.find_char('"', 1);
if (end_pos >= 0) {
include_file = include_file.substr(1, end_pos - 1);

String include_code = ShaderIncludeDB::get_built_in_include_file(include_file);
if (!include_code.is_empty()) {
// Add these lines into our parse list so we parse them as well.
Vector<String> include_lines = include_code.split("\n");

for (int j = include_lines.size() - 1; j >= 0; j--) {
lines.insert(i + 1, include_lines[j]);
}

line_count = lines.size();
} else {
// Just add it back in, this will cause a compile error to alert the user.
parsed_code += l + "\n";
}
} else {
// Include as is.
parsed_code += l + "\n";
}
} else {
// Include as is.
parsed_code += l + "\n";
}
} else {
// Include as is.
parsed_code += l + "\n";
}
}

return parsed_code;
}
53 changes: 53 additions & 0 deletions servers/rendering/shader_include_db.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/**************************************************************************/
/* shader_include_db.h */
/**************************************************************************/
/* This file is part of: */
/* GODOT ENGINE */
/* https://godotengine.org */
/**************************************************************************/
/* Copyright (c) 2014-present Godot Engine contributors (see AUTHORS.md). */
/* Copyright (c) 2007-2014 Juan Linietsky, Ariel Manzur. */
/* */
/* Permission is hereby granted, free of charge, to any person obtaining */
/* a copy of this software and associated documentation files (the */
/* "Software"), to deal in the Software without restriction, including */
/* without limitation the rights to use, copy, modify, merge, publish, */
/* distribute, sublicense, and/or sell copies of the Software, and to */
/* permit persons to whom the Software is furnished to do so, subject to */
/* the following conditions: */
/* */
/* The above copyright notice and this permission notice shall be */
/* included in all copies or substantial portions of the Software. */
/* */
/* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, */
/* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF */
/* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. */
/* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY */
/* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, */
/* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE */
/* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
/**************************************************************************/

#ifndef SHADER_INCLUDE_DB_H
#define SHADER_INCLUDE_DB_H

#include "core/object/class_db.h"

class ShaderIncludeDB : public Object {
GDCLASS(ShaderIncludeDB, Object)

private:
static HashMap<String, String> built_in_includes;

protected:
static void _bind_methods();

public:
static void register_built_in_include_file(const String &p_filename, const String &p_shader_code);
static PackedStringArray list_built_in_include_files();
static bool has_built_in_include_file(const String &p_filename);
static String get_built_in_include_file(const String &p_filename);
static String parse_include_files(const String &p_code);
};

#endif // SHADER_INCLUDE_DB_H