Conversation
| "TGI": ["GPU", "INF2"], | ||
| "TEI": ["GPU", "CPU"], | ||
| "TGILLAMACPP": ["CPU"], | ||
| "TGILLAMACPP": ["GPU", "CPU"], |
There was a problem hiding this comment.
interesting, I thought it's just a CPU image.
There was a problem hiding this comment.
Yeah and ideally for Graviton instances there should be a different container to maximize the performance, see https://huggingface.co/docs/text-generation-inference/backends/llamacpp#build-docker-image
| ENTRYPOINT ["./entrypoint.sh"] | ||
| CMD ["--json-output"] | ||
|
|
||
| LABEL dlc_major_version="2" |
There was a problem hiding this comment.
dlc_major_version = "2" ? shouldn't it be "1" considering that we are contributing it for the first time?
| ENTRYPOINT ["./entrypoint.sh"] | ||
| CMD ["--json-output"] | ||
|
|
||
| LABEL dlc_major_version="2" |
| @@ -0,0 +1,21 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
correct me if I am wrong but this is because of the vulnerability right? I'll take an AI to check with hosting platform if they have got any long term remediation for this vulnerability or not.
There was a problem hiding this comment.
my understanding is, this is an overhead every time we are gonna add a new image right?
There was a problem hiding this comment.
Indeed that's for the vulnerability, and yes this is a an overhead for all the new cuda based images
| { | ||
| "framework": "TEI", | ||
| "framework": "TGILLAMACPP", | ||
| "device": "gpu", |
There was a problem hiding this comment.
dont you want to add both CPU and GPU images?
There was a problem hiding this comment.
Yeah, tested both independently locally, but will ad that to build the two
|
just for sanity, what would be the final image tag here? |
…iner into tgi-llamacpp
…iner into tgi-llamacpp
3 participants
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.