Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add run and serve arguments for --device and --privileged #809

Merged
merged 3 commits into from
Feb 13, 2025
Merged

Add run and serve arguments for --device and --privileged #809

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b396ef8
Add run and serve arguments for --device and --privileged
cgruver Feb 13, 2025
483f229
Add --privileged and --device args to the appropriate docs
cgruver Feb 13, 2025
07bc341
Changes to satisfy code review
cgruver Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/ramalama-run.1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ path of the authentication file for OCI registries
#### **--ctx-size**, **-c**
size of the prompt context (default: 2048, 0 = loaded from model)

#### **--device**
declare host device to leak into the container

#### **--help**, **-h**
show this help message and exit

Expand All @@ -43,6 +46,9 @@ name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container

#### **--privileged**
give extended privileges to container
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

       By  default, RamaLama containers are unprivileged (=false) and cannot, for
       example, modify parts of the operating system. This is  because  by  de‐
       fault  a  container is only allowed limited access to devices. A "privi‐
       leged" container is given the same access to devices as the user launch‐
       ing the container, with the exception of virtual consoles  (/dev/tty\d+)
       when running in systemd mode (--systemd=always).

       A  privileged container turns off the security features that isolate the
       container from the host. Dropped Capabilities,  limited  devices,  read-
       only  mount points, Apparmor/SELinux separation, and Seccomp filters are
       all disabled.  Due to the disabled  security  features,  the  privileged
       field  should  almost never be set as containers can easily break out of
       confinement.

       Containers running in a user namespace (e.g., rootless containers)  can‐
       not have more privileges than the user that launched them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this, from Podman man pages.


#### **--seed**=
Specify seed rather than using random seed model interaction

Expand Down
6 changes: 6 additions & 0 deletions docs/ramalama-serve.1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ The default is TRUE. The --nocontainer option forces this option to False.

Use the `ramalama stop` command to stop the container running the served ramalama Model.

#### **--device**
declare host device to leak into the container
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add something like.

       Add a host device to the container. Optional permissions parameter  can
       be  used  to  specify device permissions by combining r for read, w for
       write, and m for mknod(2).

       Example: --device=/dev/sdc:/dev/xvdc:rwm.

The device specifiaction is passed directly to the underlying container engine.  See documentation of the supported container engine for more information.


#### **--generate**=type
Generate specified configuration format for running the AI Model as a service

Expand All @@ -70,6 +73,9 @@ set the network mode for the container
#### **--port**, **-p**
port for AI Model server to listen on

#### **--privileged**
give extended privileges to container
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto


#### **--seed**=
Specify seed rather than using random seed model interaction

Expand Down
18 changes: 17 additions & 1 deletion ramalama/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,12 @@ def configure_arguments(parser):
help="store AI Models in the specified directory",
)
parser.add_argument("-v", "--version", dest="version", action="store_true", help="show RamaLama version")
# parser.add_argument("--device",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either turn this code on or delete it.

# dest="device",
# action='append',
# type=str,
# # nargs=1,
# help="Device to leak in to the running container")


def configure_subcommands(parser):
Expand Down Expand Up @@ -801,6 +807,11 @@ def _run(parser):
default=config.get('ctx_size', 2048),
help="size of the prompt context (0 = loaded from model)",
)
parser.add_argument("--device",
dest="device",
action='append',
type=str,
help="Device to leak in to the running container")
parser.add_argument("-n", "--name", dest="name", help="name of container in which the Model will be run")
# Disable network access by default, and give the option to pass any supported network mode into
# podman if needed:
Expand All @@ -811,7 +822,12 @@ def _run(parser):
default="none",
help="set the network mode for the container",
)

parser.add_argument(
"--privileged",
dest="privileged",
action="store_true",
help="give extended privileges to container"
)
parser.add_argument("--seed", help="override random seed")
parser.add_argument(
"--temp", default=config.get('temp', "0.8"), help="temperature of the response from the AI model"
Expand Down
12 changes: 5 additions & 7 deletions ramalama/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,13 +187,11 @@ def setup_container(self, args):
if hasattr(args, "port"):
conman_args += ["-p", f"{args.port}:{args.port}"]

# Check for env var RAMALAMA_DEVICE to explicitly declare the GPU device path
device_override = 0
gpu_device = os.environ.get("RAMALAMA_DEVICE")
if gpu_device:
conman_args += ["--device", gpu_device]
device_override = 1
if device_override != 1:
if args.device:
for device_arg in args.device:
print(device_arg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like debug line.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops... yes it is... :-)

conman_args += ["--device", device_arg]
else:
if (sys.platform == "darwin" and os.path.basename(args.engine) != "docker") or os.path.exists("/dev/dri"):
conman_args += ["--device", "/dev/dri"]

Expand Down
Loading