Merge branch 'main' into dev

Author: lukvmil

.gitignore

@@ -10,4 +10,5 @@ prototypes
 dist/
 docs/
 tests/
-.rid_cache/
+.rid_cache/
+*.ndjson

README.md

@@ -401,282 +401,6 @@ node = NodeInterface(
 
 Take a look at `src/koi_net/processor/default_handlers.py` to see some more in depth examples and better understand the default node behavior.
 
-# Implementation Reference
-This section provides high level explanations of the Python implementation. More detailed explanations of methods can be found in the docstrings within the codebase itself.
-
-## Node Interface
-The node class mostly acts as a container for other classes with more specialized behavior, with special functions that should be called to start up and shut down a node. We'll take a look at each of these components in turn, but here is the class stub:
-```python
-class NodeInterface:
-    config: ConfigType
-    cache: Cache
-    identity: NodeIdentity
-    network: NetworkInterface
-    processor: ProcessorInterface
-    
-    use_kobj_processor_thread: bool
-    
-    def __init__(
-        self, 
-        config: ConfigType,
-        use_kobj_processor_thread: bool = False,
-        
-        handlers: list[KnowledgeHandler] | None = None,
-        
-        cache: Cache | None = None,
-        network: NetworkInterface | None = None,
-        processor: ProcessorInterface | None = None
-    ): ...
-
-    def start(self): ...
-    def stop(self): ...
-```
-As you can see, only a node config is required, all other fields are optional.
-
-## Node Config
-
-The node config class is a mix of configuration groups providing basic shared behavior across nodes through a standard interface. The class is implemented as Pydantic model, but provides functions to load from and save to a YAML file, the expected format within a node repo.
-
-```python
-class ServerConfig(BaseModel):
-    host: str | None = "127.0.0.1"
-    port: int | None = 8000
-    path: str | None = "/koi-net"
-    
-    @property
-    def url(self) -> str: ...
-
-class KoiNetConfig(BaseModel):
-    node_name: str
-    node_rid: KoiNetNode | None = None
-    node_profile: NodeProfile
-    
-    cache_directory_path: str | None = ".rid_cache"
-    event_queues_path: str | None = "event_queues.json"
-
-    first_contact: str | None = None
-
-class EnvConfig(BaseModel):
-    ...
-
-class NodeConfig(BaseModel):
-    server: ServerConfig | None = Field(default_factory=ServerConfig)
-    koi_net: KoiNetConfig
-
-    @classmethod
-    def load_from_yaml(
-        cls, 
-        file_path: str = "config.yaml", 
-        generate_missing: bool = True
-    ): ...
-    
-    def save_to_yaml(self): ...
-```
-
-Nodes are expected to create new node config classes inheriting from `NodeConfig`. You may want to set a default KoiNetConfig (see examples) to allow for a default config to be generated if not provided by the user. Environment variables can be handled by inheriting from the `EnvConfig` class and adding new string fields. The value of this field should be equivalent to the environment variable name, for example:
-
-```python
-class SlackEnvConfig(EnvConfig):
-    slack_bot_token: str | None = "SLACK_BOT_TOKEN"
-    slack_signing_secret: str | None = "SLACK_SIGNING_SECRET"
-    slack_app_token: str | None = "SLACK_APP_TOKEN"
-```
-
-This special config class will automatically load in the variables from the current environment, or local `.env` file. Beyond these base config classes, you are free to add your own config groups. See `config.py` in the [koi-net-slack-sensor-node](https://github.com/BlockScience/koi-net-slack-sensor-node/blob/main/slack_sensor_node/config.py) repo for a more complete example.
-
-## Node Identity
-The `NodeIdentity` class provides easy access to a node's own RID, profile, and bundle. It provides access to the following properties after initialization, accessed with `node.identity`.
-```python
-class NodeIdentity:
-    rid: KoiNetNode # an RID type
-    profile: NodeProfile
-    bundle: Bundle
-```
-This it what is initialized from the required `name` and `profile` fields in the `NodeConfig` class. Node RIDs take the form of `orn:koi-net.node:<name>+<uuid>`, and are generated on first use to the identity JSON file along with a the node profile.
-
-## Network Interface
-The `NetworkInterface` class provides access to high level network actions, and contains several other network related classes. It is accessed with `node.network`.
-```python
-class NetworkInterface:
-    graph: NetworkGraph
-    request_handler: RequestHandler
-    response_handler: ResponseHandler
-
-    def __init__(
-        self, 
-        file_path: str,
-        first_contact: str | None,
-        cache: Cache, 
-        identity: NodeIdentity
-    ): ...
-
-    def push_event_to(self, event: Event, node: KoiNetNode, flush=False): ...
-    
-    def flush_poll_queue(self, node: KoiNetNode) -> list[Event]: ...
-    def flush_webhook_queue(self, node: RID): ...
-
-    def fetch_remote_bundle(self, rid: RID): ...
-    def fetch_remote_manifest(self, rid: RID): ...
-
-    def get_state_providers(self, rid_type: RIDType): ...
-    def poll_neighbors(self) -> list[Event]: ...
-```
-
-Most of the provided functions are abstractions for KOI-net protocol actions. It also contains three lower level classes: `NetworkGraph`, `RequestHandler`, and `ResponseHandler`.
-
-### Network Graph
-The `NetworkGraph` class provides access to a graph view of the node's KOI network: all of the KOI-net node and edge objects it knows about (stored in local cache). This view allows us to query nodes that we have edges with to make networking decisions.
-```python
-class NetworkGraph:
-    dg: nx.DiGraph
-
-    def __init__(
-        self,
-        cache: Cache,
-        identity: NodeIdentity
-    ): ...
-
-    def generate(self): ...
-
-    def get_edges(
-        self, 
-        direction: Literal["in", "out"] | None = None
-    ) -> list[KoiNetEdge]: ...
-
-    def get_neighbors(
-        self,
-        direction: Literal["in", "out"] | None = None,
-        status: EdgeStatus | None = None,
-        allowed_type: RIDType | None = None
-    ) -> list[KoiNetNode]: ...
-    
-    def get_node_profile(self, rid: KoiNetNode) -> NodeProfile | None: ...
-    def get_edge_profile(
-        self,
-        rid: KoiNetEdge | None = None,
-        source: KoiNetNode | None = None,
-        target: KoiNetNode | None = None
-    ) -> EdgeProfile | None: ...
-```
-
-### Request Handler
-Handles raw API requests to other nodes through the KOI-net protocol. Accepts a node RID or direct URL as the target. Each method requires either a valid request model, or `kwargs` which will be converted to the correct model in `koi_net.protocol.api_models`.
-```python
-class RequestHandler:
-    def __init__(self, cache: Cache, graph: NetworkGraph): ...
-
-    def broadcast_events(
-        self, 
-        node: RID = None, 
-        url: str = None, 
-        req: EventsPayload | None = None,
-        **kwargs
-    ) -> None: ...
-
-    def poll_events(
-        self, 
-        node: RID = None, 
-        url: str = None, 
-        req: PollEvents | None = None,
-        **kwargs
-    ) -> EventsPayload: ...
-
-    def fetch_rids(
-        self, 
-        node: RID = None, 
-        url: str = None, 
-        req: FetchRids | None = None,
-        **kwargs
-    ) -> RidsPayload: ...
-
-    def fetch_manifests(
-        self, 
-        node: RID = None, 
-        url: str = None, 
-        req: FetchManifests | None = None,
-        **kwargs
-    ) -> ManifestsPayload: ...
-
-    def fetch_bundles(
-        self, 
-        node: RID = None, 
-        url: str = None, 
-        req: FetchBundles | None = None,
-        **kwargs
-    ) -> BundlesPayload: ...
-```
-
-### Response Handler
-Handles raw API responses to requests from other nodes through the KOI-net protocol.
-```python
-class ResponseHandler:
-    def __init__(self, cache: Cache): ...
-
-    def fetch_rids(self, req: FetchRids) -> RidsPayload:
-    def fetch_manifests(self, req: FetchManifests) -> ManifestsPayload:
-    def fetch_bundles(self, req: FetchBundles) -> BundlesPayload:
-```
-Only fetch methods are provided right now, event polling and broadcasting can be handled like this:
-```python
-def broadcast_events(req: EventsPayload) -> None:
-    for event in req.events:
-        node.processor.handle(event=event, source=KnowledgeSource.External)
-    node.processor.flush_kobj_queue()
-
-def poll_events(req: PollEvents) -> EventsPayload:
-    events = node.network.flush_poll_queue(req.rid)
-    return EventsPayload(events=events)
-```
-
-## Processor Interface
-The `ProcessorInterface` class provides access to a node's internal knowledge processing pipeline.
-```python
-class ProcessorInterface:
-    worker_thread: threading.Thread | None = None
-
-    def __init__(
-        self, 
-        cache: Cache, 
-        network: NetworkInterface,
-        identity: NodeIdentity,
-        use_kobj_processor_thread: bool,
-        default_handlers: list[KnowledgeHandler] = []
-    ): ...
-
-    def add_handler(self, handler: KnowledgeHandler): ...
-    
-    def register_handler(
-        self,
-        handler_type: HandlerType,
-        rid_types: list[RIDType] | None = None
-    ): ...
-
-    def process_kobj(self, kobj: KnowledgeObject) -> None:
-    def flush_kobj_queue(self): ...
-
-    def handle(
-        self,
-        rid: RID | None = None,
-        manifest: Manifest | None = None,
-        bundle: Bundle | None = None,
-        event: Event | None = None,
-        kobj: KnowledgeObject | None = None,
-        event_type: KnowledgeEventType = None,
-        source: KnowledgeSource = KnowledgeSource.Internal
-    ): ...
-```
-
-The `register_handler` method is a decorator which can wrap a function to create a new `KnowledgeHandler` and add it to the processing pipeline in a single step. The `add_handler` method adds an existing `KnowledgeHandler` to the processining pipeline.
-
-The most commonly used functions in this class are `handle` and `flush_kobj_queue`. The `handle` method can be called on RIDs, manifests, bundles, and events to convert them to normalized to `KnowledgeObject` instances which are then added to the processing queue. If you have enabled `use_kobj_processor_thread` then the queue will be automatically processed, otherwise you will need to regularly call `flush_kobj_queue` to process queued knolwedge objects. When calling the `handle` method, knowledge objects are marked as internally source by default. If you are handling RIDs, manifests, bundles, or events sourced from other nodes, `source` should be set to `KnowledgeSource.External`.
-
-Here is an example of how an event polling loop would be implemented using the knowledge processing pipeline:
-```python
-for event in node.network.poll_neighbors():
-    node.processor.handle(event=event, source=KnowledgeSource.External)
-node.processor.flush_kobj_queue()
-```
-
 # Development
 ## Setup
 Clone this repository: