introduce document for instructlab-sdk

[cdoern]

No description provided.

[RobotSail]

SDKs and APIs are two different things. I'm assuming here you meant SDK.

Suggested change

	Unifying these various ranges of advanced workflows under an overarching `InstructLab SDK API` will allow for new usage patterns and a clearer story on what InstructLab can and should provide as user accessible endpoints.
	Unifying these various ranges of advanced workflows under an overarching `InstructLab SDK` will allow for new usage patterns and a clearer story on what InstructLab can and should provide as user accessible endpoints.

[cdoern]

good point, folks have been colloquially calling this "The SDK API for InstructLab", but I will use just SDK to keep this simple

[RobotSail]

Yes!

[williamcaban]

should we consider core be the actual main library instead of having a subcomponent of core? Like from instructlab import InstructLab and then have things like config init as part of that initialization.

[RobotSail]

I love that you've included this as part of the dev-doc.

[cdoern]

thanks!

[RobotSail]

I feel like ilab chat today provides an unmatched experience for having a local chat with models. But we shouldn't expose this through the SDK. The reason being here is that we'd simply be wrapping around LlamaCPP or vLLM, all for a single message request. At that point, it would be faster for them to just load transformers directly and manually generate from the model.

In particular, whenever I'm testing models programatically, my workflow looks like:

1. Spin up a vLLM server with the model I want
2. Run a script with whatever pre-existing messages I've written

The script itself is super simple too, like:

import openai

client = openai.Client(
    api_base="http://localhost:8000/v1",
    api_key="empty"
)

messages = [
    {
        "role": "system",
        "content": "You are an AI assistant"
    },
    {
        "role": "user",
        "content": "Why is Linux better than Windows?"
    }
]

response = client.chat.completions.create(messages, model="my-model")
assistant_response = response.choices[0].content

So if you're already at the level of consuming an SDK, I feel like it's a very simple task, so exposing it might not give much benefit.

[RobotSail]

This is a good document overall and a step in the right direction.

[nimbinatus]

Just one question on my end :) And it may be too far out of scope; feel free to ignore it!

[nimbinatus]

As this structure is what the current CLI has, I may be asking a larger question and won't be offended if you shut this as resolved to handle it somewhere else. Would it make more sense for the .taxonomy.diff tree to be under .data, to eventually leave room for alternative forms of data sorting and ingestion into the pipeline (i.e., is taxonomy a subfield of data management overall, with SDG as another subfield)?

[cdoern]

I think grouping taxonomy utilities under the general InstructLab class makes the most sense. I have updated the doc with a new design, lmk what you think

[jwm4]

This seems like a good idea. A lot of work though, of course.

[nathan-weinberg]

Overall looking good to me, a few comments

[anastasds]

Is this really an SDK? Is this not just systematization of interfaces in Python modules, that is, Python APIs? For example, there is no "SDK" artifact to download.

This is actually about cleanly modularizing the codebase, isn't it?

[anastasds]

Having *.api.* submodules makes this seems a bit like defining interfaces, which isn't really a pattern in Python. What does having .api. subpackages provide? Would they provide some sort of stability guarantee?

[cdoern]

I am actually thinking of renaming this .core.... since this isn't an API.

Grouping the code that we want folks to publicly consume is a common trend

[cdoern]

@anastasds:

Is this really an SDK? Is this not just systematization of interfaces in Python modules, that is, Python APIs? For example, there is no "SDK" artifact to download.
This is actually about cleanly modularizing the codebase, isn't it?

The goal of this is to have folks who are currently haphazardly consuming different parts of InstructLab to have a centralized "core" of code that is versioned and static. What we call it, I don't feel strongly about, but it is pretty close to an SDK I believe. If there is some sort of standardization we need to fall into, we can. but other projects like llama stack do something similar but they do package it separately.

We could scope packaging it separately but having it live within the instructlab repo.

[williamcaban]

should we consider core be the actual main library instead of having a subcomponent of core? Like from instructlab import InstructLab and then have things like config init as part of that initialization.

[williamcaban]

The intent with the Python SDK should be to serve the Data Scientists and AI engineers doing experiments as the primary use. Then the constructs required for embedding or extending InstructLab from other products and platforms.

For a data science experience I propose to be inspired to what sklearn or keras or pytorch high level structure provide for this persona.

For example, what would it take to have an experience similar to :

from instructlab import InstructLab, SDG, Train, Eval

ilab = InstructLab(taxonomy=<path_to_taxonomy>) # and on initizalization, the taxonomy is diff & loaded

# like in Pandas DataFrame.describe()
ilab.describe() # prints a summary of the taxonomy diff and attributes
ilab_df = ilab.data.ingest() # ingest the data from the taxonomy and return a dataset as a Pandas DataFrame

oa_client = {
    "url": "https://api.openactive.io/v1",
    "api_key": "YOUR_API_KEY"
}

##
# SDG interactions
##
ilab_sdg=SDG(client=oa_client,
             data=ilab_df,
             teacher_model={<model_and_attributes>},
             ) # ilab SDG class

ilab_sdg.load_pipeline(<path_to_pipeline>) # load a pipeline from a file

for block in ilab_sdg.pipeline.get_blocks():
    # do block customization
    ilab_sdg.pipeline[block].something()
    ilab_sdg.pipeline[block].block.foo = "bar"

ilab_sdg.run(
    callback=<callback_function>, # a function to call after executing each block (e.g. to report progress, or save intermediate results)
) # generate the SDG data

##
# Training interactions
##
ilab_train = Train(dataset=ilab_sdg.data, 
                   student_model={<model_and_attributes>},
                     ) # ilab Train class

ilab_train.run(
    callback=<callback_function>, # a function to call after each cycle/teration/epoch loop is run (e.g. to report progress, or save intermediate results, or to stop the training if certain stopping criteria are met)
) # execute the Training loop

ilab_train.model_save(<path_to_save>, resolution={}, model_format={gguf|safe_tensors|onnx|etc}) # save the model to a file in the specified format

##
# Evaluation
###

ilab_eval = Eval(dataset=[<path_to_eval_dataset>], 
                 endpoints=[<path_to_endpoints_for_eval>], 
                 ) # ilab Eval class

ilab_eval.evals(
    [list_of_evaluations], # a list of evaluations to run (e.g. accuracy, precision, dk-bench, etc.)
)

ilab_eval.run(
    callback=<callback_function>, # a function to call after each evaluation is run (e.g. to report progress, or save intermediate results)
)

ilab_eval.summary() # print a summary of the evaluation results

ilab_eval.save(<path_to_save>, format={jsonl|parquet|csv|xls}) # save the evaluation results to a file

ilab_eval.export_report(<path_to_save>, format={html|pdf}) # report with  the evaluation results

I would propose to focus on accelerating the goal of the persona (the data scientists), and not on mapping or exposing the InstructLab internal architecture.

[cdoern]

I think this is possibly a good end goal, but would require significant re-architecture of InstructLab, and the libraries (what they expose).

For the purposes of this dev-doc, I can incorporate some of this into what I am proposing, but for an alpha SDK, keeping the interactions as simple yet expandable as possible is my goal.

So we should aim to not require library adjustments/changes at first, and then we can add functionality once the structure is in place.

Let me incorporate some of this and I will update the PR

[alimaredia]

I think @williamcaban has a point here about the persona we're building this SDK for. Who are we making this SDK for? Datascientist? People trying to build REST-API based instructlab services? The design of the SDK might vary based on who our target user of the SDK would be.

[bbrowning]

It's definitely important to understand the persona and goal of the SDK. William is proposing a fairly large re-architecting of the APIs, while this dev doc seems mostly focused on exposing existing CLI flows and functionality via a Python SDK. Is our goal to expose the existing end-to-end flow via a Python SDK? Or to provide a new way to interact with various InstructLab components that's more granular and flexible in how you compose your own end-to-end workflow from the pieces we're exposing?

[cdoern]

I am making some updates to the PR (some are already up), aiming to meld the two approaches a bit, lmk what you think

[bbrowning]

I think the updates look like a reasonable enough place for work to start.

The actual Python code and list of APIs is only illustrative, right? From reading this, it's my understanding that the actual parameters and methods on individual classes shown here are just an example since you call out future work is to design the actual SDK based on the structure above and negotiate user contracts with library maintainers. For example, if SDG points out that the Data.Generator constructor needs different params or that we have to actually use the taxonomy diff results as input to data generation as opposed to just checking if there is a diff, this kind of thing can be ironed out later?

[nathan-weinberg]

One additional point I want to raise - what we definitely don't want is to be exposing pure library functionalities through a class in the Core repo - users that want that should just be importing the libraries directly.

As Charlie notes above what we want here is an SDK for the opinionated ilab workflow that this package is centered around - for example, our Data class wouldn't expose every public function in the SDG library, but it would expose functions we have within src/instructlab/data such as generate_data and list_data

[booxter]

This is a great idea. We should incorporate real experience from current consumers into the design of the Python public API.

[booxter]

probably obvious to you but consumers would be better served if we refer and publish the SDK as "Python" SDK (to contrast with whatever other bindings we could have in the future).

[booxter]

What's the mechanism to collect these real use patterns from external consumers, to make sure their needs are properly served?

[cdoern]

good question. Most of our usage information is coming from internal use cases combined with reports of what people are asking for colloquially. We have no request gathering system as of yet

[booxter]

Understood. I'd advise then to at least have several interview sessions with potential consumers, collecting requests / feedback / pain points.

[booxter]

(Thinking while reading) "an opinionated" (but perhaps LESS opinionated?) flow.

[booxter]

within the bounds of what the Libraries expose as public APIs

What's the current mechanism in which libraries communicate which APIs are public and which are not? Or is the definition of a "public API" here, loosely, "whatever ilab already pulls from the library"? Or, alternatively, that will be the result of "contract negotiation" that you mention below?

[cdoern]

most libraries have made APIs over time and are just now standardizing what they want public vs private. Part of the process of building this SDK will be making final decisions on that front.

In regard to opinionated, yes this will be less opinionated and more user driven

[nathan-weinberg]

I think the CLI will always be more "opinionated" than the SDK - in terms of functionality, we're exposing the same functionality to both "consumers" but the nature of the different interfaces will likely always allow for more flexibility with the SDK (since it's easier to orchestrate Python classes than CLI commands)

[booxter]

Not an issue with the proposal itself since the snippet is only illustrative, but I'd consider whether ilab SDK should promote using file paths for any artifacts. I'd rather have ilab encapsulate the knowledge about storage options (for models and datasets) and then allow SDK users to refer to the artifacts by their names. (This would be in line with the spirit of instructlab/instructlab#1832 and some other issues under instructlab/instructlab#1871)

[booxter]

api -> core? (though perhaps "api" is a better name for a module that is meant to be consumed externally. That said, API may also refer to upcoming REST API initiative, so perhaps it's better to avoid this word either. public or sdk would be another option with clear intent.) Whatever the final name, these should be unified to use just one across the doc.

[nathan-weinberg]

core is my preference, personally

[alimaredia]

If I understand this right src/instructlab/data/generate.py would just kick off processes using the code in api/data/generate.py? Is there a reason that couldn't be done in cli/data/generate.py? Is there a reason we need to go from 3 files -> 4 to do the same thing?

[nathan-weinberg]

@cdoern talked about this in-person - basically this decision boils down to whether or not we need process management or not for the SDK

[anastasds]

@cdoern I want to revisit and re-emphasize the point I raised earlier.

I really do not see how this is an SDK. This is about Python API design and modularization.

The act of logically grouping operations, giving them good names, and understandable function signatures - of defining the "nouns" and "verbs" in the application - is an act of... wait for it... domain modeling!

The work being proposed here is fundamentally a modeling and modularization effort - it is not about creating an SDK.
(And I fully support it!)

[nathan-weinberg]

To illustrate my point above, I would prefer

Suggested change

	instructlab.core.InstructLab
	instructlab.core.InstructLab.taxonomy.diff
	instructlab.core.InstructLab.system.info
	instructlab.core.InstructLab.config.show
	instructlab.core.InstructLab.config.get
	instructlab.core.Taxonomy.diff
	instructlab.core.System.info
	instructlab.core.Config.show
	instructlab.core.Config.get

[JamesKunstle]

This could be accomplished w/ __init__.py module visibility too.

[cdoern]

just as a note, these classes are departing from the structure of the other ones where its Model.Trainer.train_model for example, is this something we are ok with?

[nathan-weinberg]

Is this a departure from that? I don't see how

[nathan-weinberg]

One additional point I want to raise - what we definitely don't want is to be exposing pure library functionalities through a class in the Core repo - users that want that should just be importing the libraries directly.

As Charlie notes above what we want here is an SDK for the opinionated ilab workflow that this package is centered around - for example, our Data class wouldn't expose every public function in the SDG library, but it would expose functions we have within src/instructlab/data such as generate_data and list_data

[nathan-weinberg]

Suggested change

These endpoints in combination with the curated InstructLab Config File will open up these workflows to users and allow InstructLab to be easily incorporated into other projects. Allowing people to run things like data generation, and full fine-tuning via an SDK that pulls in their pre-existing config.yaml but also can be run independently will open new avenues for InstructLab adoption and extensibility.

These endpoints in combination with the curated InstructLab Config File will open up these workflows to users and allow InstructLab to be easily incorporated into other projects. Allowing people to run things like data generation, and full fine-tuning via an SDK that pulls in their pre-existing `config.yaml` but also can be run independently will open new avenues for InstructLab adoption and extensibility.

[nathan-weinberg]

Would it not make more sense for both the CLI and "Core" to be consuming from the generation and process logic separately? Why couple the CLI with the SDK at all?

[JamesKunstle]

It's possible we'd want this in some situations, but I think it's a good design decision to plop the CLI on top of the SDK as much as possible. For one thing, it minimizes duplicate code doing similar things. Another good thing is that we'd be testing our own SDK by wrapping it- testing the CLI would also be (only slightly indirectly) testing the SDK.

[booxter]

I don't think core should consume process management at all. Vice versa, CLI - through process management layer, if needed - will consume SDK. The user of the SDK should not be required to use our particular process management model. (Nor I think process management module is interesting to general consumers per se as a separate library. Prefer keeping it private to ilab CLI flow.)

[nathan-weinberg]

Hmmm it's an interesting point @JamesKunstle @booxter - but isn't the difference here that the CLI and the SDK are consuming the same logic but exposing it in different ways, i.e. via the command line versus importable classes? I suppose the CLI could utilize the SDK classes, but at that point it seems to me that the logic itself could simply expose importable classes and that would cover the usecase?

Indeed the process management piece shapes a lot of this conversation

[booxter]

As far as I understood so far, process management is applicable where we anticipate long running processes (generate, train etc.) Some commands are like that and will go through the chain of calls as:

click -> (cli handler) - ...schedules process... -> process management -> ...spawns process... -> (process handler) -> sdk -> libraries

while for processes that don't need background execution, it would be:

click -> (cli handler) -> sdk -> libraries

In the scheme above, sdk never consumes process management (because it's not expressed in terms of separate async processes).

[JamesKunstle]

It'd be really cool if we could use the vLLM offline inference features in an SDK scenario like this. My understanding is that that's the highest throughput calling pattern if requests are well-organized to begin with.

[JamesKunstle]

I really like the design changes represented in this document.
One consideration I'd raise as we proceed is that we may want to avoid the object-oriented pattern that the HF Trainer object is representative of, and prefer a functional programming convention.
The ravioli-code "Trainer" / "Generator" classes are a problematic design pattern elsewhere in this problem space. instructlab/training has avoided duplicating this convention because it's harder to reason about and encourages large procedural instance methods.

Instead, it's more readable, hackable, and composable to have an SDK expose functional building blocks w/ attention to the interfaces informing the call.

For illustration, why ever use:

trainer = Trainer(
    data=data,
    model=model,
    epochs=5,
    lr=1e-4,
    ...
    )

trainer.train()

when one could just run:

train(
    data=data,
    model=model,
    epochs=5,
    lr=1e-4,
    ...
    )

[booxter]

Agreed with @JamesKunstle on NOT adopting object oriented primitives to describe process flow (ilab methodology being a pipeline of processes with - hopefully pure - inputs and outputs). (As an anecdote: We have Java-style RAG implementation in tree with factories and all, which is very hard to follow and unravel due to all the layers of indirection, and I don't think this is a good pattern to continue in this code base, esp. for "SDK".)

---

@anastasds I wonder what makes something an SDK (SDK being Software Development Kit) if not a "pack of compute primitives meant to be consumed externally", which this proposal is (?) (Obviously, proper modeling is part of any good API and SDK design; but this seems wider.)

AFAIU a primary problem being solved here "current consumers of InstructLab are finding themselves importing different library private and public APIs", which is being solved by giving these consumers something to consume that is meant to be consumed. This is SDK.

[anastasds]

@booxter in that general of a sense, every source codebase is an SDK.

An sdk package that contains all functions, classes, constants, etc meant to be used for custom development would reasonably be called an SDK. It would unarguably be an SDK if it were its own additional artifact that could be optionally downloaded by those intending to develop custom functionality, e.g. pip install instructlab[sdk].

Having *.api.* or *.core.* submodules interspersed throughout is just Python APIs.

[cdoern]

@booxter @JamesKunstle given the consumption patterns of who is going to use this while also trying to the best of our ability to align with InstructLab's core values here is what I propose:

rather than a "spaghetti" structure like:

from instructlab.core import Data

sdg_client = Data.Generator(<args>)
sdg_client.generate()

I propose instead to at least do

from instructlab.core import Data, Config


config_object = Config.init(auto_detect=True)
data_client = Data(taxonomy_path="", cfg=config_object)
data_client.generate_data()

I think this structure at a minimum is what we need for the following reasons:

1. "parent" classes align well with the ethos of the command line structure. a Data class can be instantiated with common values meant to be inherited by other commands so args are not necessary each time
2. current consumers expect a hierarchical structure of InstructLab, rather than doing something like from instructlab.core import generate_data
3. adding a level of abstraction allows the codebase to meld better with this new "core" package. Alternatively moving key functions into a core package at a top level would make the codebase confusing to understand as opposed to being grouped under a class like Data

In general usability, searchability (is that a word?), and our inheritance structure would suffer from not having at least top level cmd classes.

@nathan-weinberg do you agree with the above?

[nathan-weinberg]

@booxter @JamesKunstle given the consumption patterns of who is going to use this while also trying to the best of our ability to align with InstructLab's core values here is what I propose:

rather than a "spaghetti" structure like:

from instructlab.core import Data

sdg_client = Data.Generator(<args>)
sdg_client.generate()

I propose instead to at least do

from instructlab.core import Data, Config


config_object = Config.init(auto_detect=True)
data_client = Data(taxonomy_path="", cfg=config_object)
data_client.generate_data()

I think this structure at a minimum is what we need for the following reasons:

1. "parent" classes align well with the ethos of the command line structure. a Data class can be instantiated with common values meant to be inherited by other commands so args are not necessary each time

2. current consumers expect a hierarchical structure of InstructLab, rather than doing something like `from instructlab.core import generate_data`

3. adding a level of abstraction allows the codebase to meld better with this new "core" package. Alternatively moving key functions into a core package at a top level would make the codebase confusing to understand as opposed to being grouped under a class like `Data`

In general usability, searchability (is that a word?), and our inheritance structure would suffer from not having at least top level cmd classes.

@nathan-weinberg do you agree with the above?

Indeed - I think from a usability perspective it makes sense that the SDK classes would align with the CLI command groups - ultimately we're exposing the same (if not more) functionality through a different interface, but keeping the naming conventions of our original interface (the CLI) is a good thing IMO.

I also echo what @JamesKunstle says above about having sub-classes like trainers and generators etc. etc. - it's overengineering in my opinion, though I could be convinced otherwise given a realistic example.

[booxter]

+1. Keeping the existing click command group hierarchy reflected in "SDK" while avoiding popping up classes is a good balance. Represent entities with objects and processes with methods / functions (Not with a "factory" to execute processes or to produce objects).

---

On the question of whether something is SDK or not - the main point is the intent. If the intent is for external consumers to take it and integrate with it, then it's SDK. Whether it's shipped in the same artifact is an implementation detail.

That said, as long as the name for this "thing" is not just "API" (which will be ambiguous for upcoming REST API), but e.g. "Python API" then I am ok with renaming too.

[booxter]

This is correctly directed. We can flush out details (verbiage etc.) during development phase. What I'd encourage is to always refer to this as Python API/SDK going forward.

[booxter]

The actual groups are

Commands:
  config    Command group for interacting with the config of InstructLab.
  data      Command group for interacting with the data generated by...
  model     Command group for interacting with the models in InstructLab.
  process   Command group for interacting with the processes run by...
  rag       Command group for interacting with the RAG for InstructLab.
  system    Command group for all system-related command calls.
  taxonomy  Command group for interacting with the taxonomy of InstructLab.

I don't think Eval (process) makes sense a class. What makes sense is Model.eval() though.

Note each click command group is tied to a "thing" (object) (except RAG? Which perhaps should've moved under data?), not a verb.

---

This is probably just an oversight though since I don't see Eval listed in the examples below, only in the line 60.

[booxter]

I wonder how you distinguish between parameters passed into verb methods (client, data here) and parameters passed to class.__init__ (line 100).

[booxter]

Good goal. Eventually we can programmatically enforce that no imports for libraries are happening except under the sdk package.