Larger models available or planned?

[jaggzh]

Anything larger than 2.7B cooking? I'm itching to test the larger capacity and its scaling against the larger small LLMs of comparable size (or comparable resource use).

[ridgerchu]

Hi, sorry that until now we still do not have enough resources to scale up our model... Thanks for your interest!

[Ovid]

@ridgerchu Has there been any movement here that you can publicly discuss? This work seems significant enough in terms of environmental impact that it seems like many companies would be very interested in pushing this forward.

[ridgerchu]

@Ovid Unfortunately, as of now, we haven't been able to secure a feasible sponsor that would allow us to scale this work😢 Despite the potential environmental impact and efficiency gains, finding the right support to move this research forward has been challenging. We're continuing to explore opportunities and remain hopeful about the future prospects of this technology...

[Melaron]

@ridgerchu, that is surprising to hear. When I read the article, I thought that there would be a rush to utilize this — it seems quiet so far. I hope you can find a good sponsor soon.

[ridgerchu]

Thank you all for your interest! We have found some interesting companies or organizations that may fund us to scale up. Although negotiations are still ongoing, we're hopeful that we can develop at least a Mistral 7B level model.

[Ovid]

@ridgerchu Happy New Year!

Any news on funding you can share?

[ridgerchu]

Happy New Year @Ovid !

Actually we are starting trying to scale up the model, we first step may start with a medium-size model with near-SOTA performance, if it successfully scale we will also open source it!

[impredicative]

Will this matmulfreellm preserve efficiency gains on top of bitnet, e.g. https://github.com/microsoft/BitNet and https://github.com/kyegomez/BitNet ? If it does, then someone like Sandia could perhaps help with funding.