Install and Test Stanford Alpaca on Mac

Performance of Alpaca included

From Standford alpaca

Problem: Why do I use Alpaca

OpenAI API costs money and I don’t want to pay. I have many problems using hugging face models because of M1 incompatibility. Hence, I decided to download an LLM model to my local and test it there. I pick Stanford Alpaca because it says that it is a model combining llama and GPT, and it’s free!

Installation

I follow the Alpaca GitHub instruction.

git clone https://github.com/antimatter15/alpaca.cpp
cd alpaca.cpp

make chat
./chat

And then download the ggml-alpaca-7b-q4.bin and place it in the same folder as the chat. The size of the alpaca is 4 GB. It’s not skinny.

You can also follow through Dalai GitHub just remember to change the installation command to alpaca (not llama because llama is so fat).

Run the code

You can run the code through this command in your terminal.

cd alpaca/
cd ../alpaca/    
./chat_mac

Problems with Alpaca on Mac

1. This model is very slow at producing text, which may be due to my Mac’s performance or the model’s performance. It also slows down my entire Mac, possibly due to RAM limitations.

2. The results are disappointing. They claim that Alpaca is a high school student while GPT 3.5 is a college student. However, I find that Alpaca gives me very questionable answers, especially when I try to write about Tim Urban. The results related to Tim are bad. I think the reason for this is that GPT-4 has 100 trillion parameters, while the model I am using only has 7 billion.

This text takes me more than 10 minutes and I cannot wait anymore.

3. I cannot integrate it with Langchain. I can successfully run it on my terminal but when I try to load the model on my Python notebook, it always throws an error.

(tim-urban-env) swimmingcircle@Esthers-MacBook-Pro tim_urban_writing % python3 alpaca-test.py
llama.cpp: loading model from ./ggml-alpaca-7b-q4.bin
error loading model: unexpectedly reached end of file
llama_init_from_file: failed to load model
Traceback (most recent call last):
  File "/Users/swimmingcircle/Code/tim_urban_writing/alpaca-test.py", line 2, in <module>
    embeddings = LlamaCppEmbeddings(model_path="./ggml-alpaca-7b-q4.bin")
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCppEmbeddings
__root__
  Could not load Llama model from path: ./ggml-alpaca-7b-q4.bin. Received error  (type=value_error)

Conclusion: I don’t recommend Alpaca.

I decide giving up using Alpaca because I don’t think it was worth the effort for figuring out the latency, langchain integration problems when the results are meh.