Genie Discord forum

Author AvatarSimone Gabbriellini
11/28/2023, 8:26:51 PM

coming from Flask, I am not sure here how to load something (an LLM) when the API is launched... It cannot be handled in the route otherwise every call will hit that, but I am not sure where I should load my model... any hint to documentation that I might have missed is more than welcome!

Author AvatarPere
11/29/2023, 11:32:43 AM

You can load it when you load the API module, before the route definitions. This is how I'm doing it in an app I'm writing now (see image)

see the includeand @loadstatements. You could have something like

module API
LLM = load_my_llm()
route("/llm") do
Author AvatarSimone Gabbriellini
11/29/2023, 12:16:41 PM

thank you! it is indeed simpler than I thought...

Author Avataressenciary
12/5/2023, 4:20:03 PM

beware of globals though - make it a const or a Ref

Author AvatarSimone Gabbriellini
12/6/2023, 8:18:07 PM

mm not sure I understand, I am really a newbie in julia