大模型部署手记（15）LLaMa2+Ubuntu18.04+privateGPT_privategpt llama2

1.简介：

组织机构：Meta（Facebook）

代码仓：https://github.com/facebookresearch/llama  https://github.com/ymcui/Chinese-LLaMA-Alpaca-2

模型：llama-2-7b sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

下载：使用download.sh下载或Huggingface.co或aliendao.cn下载

硬件环境：暗影精灵7Plus

Ubuntu版本：18.04

内存 32G

GPU显卡：Nvidia GTX 3080 Laptop （16G）

2.代码和模型下载

参考 大模型部署手记（7）LLaMA2+Jetson AGX Orin-云社区-华为云 下载 llama-2-7b模型。

将下载好的模型文件拷贝到 ~/privateGPT/models 目录下：

cd ~

git clone https://github.com/imartinez/privateGPT

3.安装依赖

conda create -n private310 python=3.10 -y

conda activate private310

cd privateGPT

先安装torch支持CUDA11.8版本：

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

pip3 install -r requirements.txt

安装Poetry：

curl -sSL https://install.python-poetry.org | python3 -

cd ~/privateGPT

poetry install

查看：https://github.com/python-poetry/poetry/issues/6354

照此执行：

export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

poetry install

poetry shell

感觉它是要重新进入conda环境

conda activate private310

4.部署验证

ls source_documents

加载模型试试：

python

from llama_cpp import Llama

llm = Llama(model_path="./models/7B/ggml-model-q4_0.gguf")

开始解析文本：

python ingest.py

pip install sentence_transformers

再来：

python ingest.py

它应该是想从Huggingface.co下载 sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 这个模型，显然这个是不行的。

打开 https://www.sbert.net/docs/pretrained_models.html​

下载地址如下：https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

想办法去下载吧！

将其下载到windows，再上传到ubuntu的 ~/privateGPT/models目录：

修改.env:

EMBEDDINGS_MODEL_NAME=./models/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

python ingest.py

mkdir -p ./models/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1_Pooling

cd ./models/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/

cp * 1_Pooling/

cd ~/privateGPT

python ingest.py

(private310) zhanghui@zhanghui-OMEN-by-HP-Laptop-17-ck0xxx:~/privateGPT$ python ingest.py
Traceback (most recent call last):
  File "/home/zhanghui/privateGPT/ingest.py", line 185, in <module>
    main()
  File "/home/zhanghui/privateGPT/ingest.py", line 155, in main
    embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)
  File "/home/zhanghui/anaconda3/envs/private310/lib/python3.10/site-packages/langchain/embeddings/huggingface.py", line 64, in __init__
    self.client = sentence_transformers.SentenceTransformer(
  File "/home/zhanghui/anaconda3/envs/private310/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 95, in __init__
    modules = self._load_sbert_model(model_path)
  File "/home/zhanghui/anaconda3/envs/private310/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 840, in _load_sbert_model
    module = module_class.load(os.path.join(model_path, module_config['path']))
  File "/home/zhanghui/anaconda3/envs/private310/lib/python3.10/site-packages/sentence_transformers/models/Pooling.py", line 120, in load
    return Pooling(**config)
TypeError: Pooling.__init__() got an unexpected keyword argument '_name_or_path'

在python命令行试试：

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('./models/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

仔细看看，果然有个1_Pooling目录，下载config.json

{
  "word_embedding_dimension": 384,
  "pooling_mode_cls_token": false,
  "pooling_mode_mean_tokens": true,
  "pooling_mode_max_tokens": false,
  "pooling_mode_mean_sqrt_len_tokens": false
}

cd ./models/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1_Pooling

rm -f *

vi config.json 将上述文本复制进去。

cd ~/privateGPT

再来：

python ingest.py

分析本地文件成功了！

此时会生成一个db目录：

执行推理：

python privateGPT.py

who is Intel’s CEO?

查看 ​www.cnblogs.com/leon-dal/p/17639852.html

修改privateGPT.py

将：

llm = LlamaCpp(model_path=model_path, max_tokens=model_n_ctx, n_batch=model_n_batch, callbacks=callbacks, verbose=False)

改为：

llm = LlamaCpp(model_path=model_path, max_tokens=model_n_ctx, n_batch=model_n_batch, callbacks=callbacks, verbose=False, n_ctx=model_n_ctx)

执行：

python privateGPT.py

看看 分析前的原文：state_of_the_union.txt

再问一个问题：

How much is their investment?

这段话回答得有点怪。。

最后看下执行的全文：

(private310) zhanghui@zhanghui-OMEN-by-HP-Laptop-17-ck0xxx:~/privateGPT$ python privateGPT.py
 
Enter a query: who is Intel’s CEO?
 Pat Gelsinger (in the middle of the picture)
\end{blockquote}
 
In short, I guess I'm trying to find out if the quote above can be considered validated by the context in which it appears.
Comment: "Intel’s CEO, Pat Gelsinger"
Answer: It is a rhetorical question.  The answer is that he does not know.  If he knew who Intel's CEO was, then why would he say "I don't know", unless he thought it was someone else?
 
> Question:
who is Intel’s CEO?
 
> Answer (took 41.18 s.):
 Pat Gelsinger (in the middle of the picture)
\end{blockquote}
 
In short, I guess I'm trying to find out if the quote above can be considered validated by the context in which it appears.
Comment: "Intel’s CEO, Pat Gelsinger"
Answer: It is a rhetorical question.  The answer is that he does not know.  If he knew who Intel's CEO was, then why would he say "I don't know", unless he thought it was someone else?
 
> source_documents/state_of_the_union.txt:
That would be one of the biggest investments in manufacturing in American history.
 
And all they’re waiting for is for you to pass this bill.
 
So let’s not wait any longer. Send it to my desk. I’ll sign it.
 
And we will really take off.
 
And Intel is not alone.
 
There’s something happening in America.
 
Just look around and you’ll see an amazing story.
 
The rebirth of the pride that comes from stamping products “Made In America.” The revitalization of American manufacturing.
 
> source_documents/state_of_the_union.txt:
Up to eight state-of-the-art factories in one place. 10,000 new good-paying jobs.
 
Some of the most sophisticated manufacturing in the world to make computer chips the size of a fingertip that power the world and our everyday lives.
 
Smartphones. The Internet. Technology we have yet to invent.
 
But that’s just the beginning.
 
Intel’s CEO, Pat Gelsinger, who is here tonight, told me they are ready to increase their investment from
$20 billion to $100 billion.
 
> source_documents/state_of_the_union.txt:
Let me give you one example of why it’s so important to pass it.
 
If you travel 20 miles east of Columbus, Ohio, you’ll find 1,000 empty acres of land.
 
It won’t look like much, but if you stop and look closely, you’ll see a “Field of dreams,” the ground on which America’s future will be built.
 
This is where Intel, the American company that helped build Silicon Valley, is going to build its $20 billion semiconductor “mega site”.
 
> source_documents/state_of_the_union.txt:
But in my administration, the watchdogs have been welcomed back.
 
We’re going after the criminals who stole billions in relief money meant for small businesses and millions of Americans.
 
And tonight, I’m announcing that the Justice Department will name a chief prosecutor for pandemic fraud.
 
By the end of this year, the deficit will be down to less than half what it was before I took office.
 
The only president ever to cut the deficit by more than one trillion dollars in a single year.
 
Enter a query: How much is their investment?
 
 
Answer:
 
It’s not the kind of thing you can see right now or measure with a ruler.
 
We are restoring our economy from scratch, which was about to collapse under Obama.
 
With the help of the American people, we have already turned around this economy.
 
These are jobs that didn’t even exist 30 years ago. We are making history again!
 
We’ve begun a new industrial revolution – one that is built on innovation and technology, but most importantly, it is American made.
 
This is the next chapter in our story:
 
An economy that works for everyone. An America that truly is great again.
 
> Question:
How much is their investment?
 
> Answer (took 41.08 s.):
 
 
Answer:
 
It’s not the kind of thing you can see right now or measure with a ruler.
 
We are restoring our economy from scratch, which was about to collapse under Obama.
 
With the help of the American people, we have already turned around this economy.
 
These are jobs that didn’t even exist 30 years ago. We are making history again!
 
We’ve begun a new industrial revolution – one that is built on innovation and technology, but most importantly, it is American made.
 
This is the next chapter in our story:
 
An economy that works for everyone. An America that truly is great again.
 
> source_documents/state_of_the_union.txt:
During the pandemic, these foreign-owned companies raised prices by as much as 1,000% and made record profits.
 
Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers.
 
And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.
 
That ends on my watch.
 
Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect.
 
> source_documents/state_of_the_union.txt:
Up to eight state-of-the-art factories in one place. 10,000 new good-paying jobs.
 
Some of the most sophisticated manufacturing in the world to make computer chips the size of a fingertip that power the world and our everyday lives.
 
Smartphones. The Internet. Technology we have yet to invent.
 
But that’s just the beginning.
 
Intel’s CEO, Pat Gelsinger, who is here tonight, told me they are ready to increase their investment from
$20 billion to $100 billion.
 
> source_documents/state_of_the_union.txt:
When we use taxpayer dollars to rebuild America – we are going to Buy American: buy American products to support American jobs.
 
The federal government spends about $600 Billion a year to keep the country safe and secure.
 
There’s been a law on the books for almost a century
to make sure taxpayers’ dollars support American jobs and businesses.
 
Every Administration says they’ll do it, but we are actually doing it.
 
> source_documents/state_of_the_union.txt:
Companies are choosing to build new factories here, when just a few years ago, they would have built them overseas.
 
That’s what is happening. Ford is investing $11 billion to build electric vehicles, creating 11,000 jobs across the country.
 
GM is making the largest investment in its history—$7 billion to build electric vehicles, creating 4,000 jobs in Michigan.
 
All told, we created 369,000 new manufacturing jobs in America just last year.
 
Enter a query:

英文推理之后，中文怎么搞，下次再说。