meta-llama

/

Llama-3.1-8B-Instruct

like 2.97k

Follow

Meta Llama 7,289

First: is there are search feature for this forum? Or do I just used the general search at the top of th page?
Second: I have installed 8B-Instruct on my Fedora 38 server. I hacked one of the script examples and I am interacting with it on the command line (my prefered environment for coding or administration). I have 32 GB of RAM installed and I was told this package required 16. Here is my current top output while the script is in a loop waiting for me to input a question:
top - 08:38:24 up 3 days, 20:10, 2 users, load average: 0.02, 0.57, 0.62
Tasks: 261 total, 1 running, 260 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 31.2 total, 10.2 free, 1.0 used, 20.1 buff/cache
GiB Swap: 8.0 total, 7.9 free, 0.1 used. 29.8 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

30813 root 20 0 9104 3760 2864 R 0.7 0.0 0:03.76 top

Now, Here is my the output after I submit the question:

top - 08:40:23 up 3 days, 20:12, 2 users, load average: 0.29, 0.45, 0.57
Tasks: 262 total, 2 running, 260 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.4 us, 0.1 sy, 0.0 ni, 91.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 31.2 total, 10.2 free, 1.0 used, 20.1 buff/cache
GiB Swap: 8.0 total, 7.9 free, 0.1 used. 29.8 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

30600 root 20 0 20.0g 14.6g 14.2g R 100.0 46.7 16:49.29 pt_main_thread
18875 systemd+ 20 0 15224 1808 1424 S 0.3 0.0 3:49.99 systemd-oomd
28103 root 20 0 0 0 0 I 0.3 0.0 0:53.45 kworker/11:3-events

The responses are taking an average of 9 minutes to process, yet, as you can see, I'm barely using the resources I have.
I approached this as 8B-Instruct config issue at first and managed to shave off a couple of minutes from the original 12 to 13 minutes by making some minor changes to the parameters set in my script, but I've maxed out those options and I'm wondering is this an environment issue?
HERE IS MY ENVIRONMENT
SHELL=/bin/bash
HISTCONTROL=ignoredups
HISTSIZE=1000
HOSTNAME=richard.me
EDITOR=/usr/bin/vim
PWD=/llama/llama-models-main
LOGNAME=root
HOME=/root
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.avif=01;35:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.webp=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.m4a=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.oga=01;36:.opus=01;36:.spx=01;36:.xspf=01;36:~=00;90:#=00;90:.bak=00;90:.old=00;90:.orig=00;90:.part=00;90:.rej=00;90:.swp=00;90:.tmp=00;90:.dpkg-dist=00;90:.dpkg-old=00;90:.ucf-dist=00;90:.ucf-new=00;90:.ucf-old=00;90:.rpmnew=00;90:.rpmorig=00;90:.rpmsave=00;90:
TERM=xterm
LESSOPEN=||/usr/bin/lesspipe.sh %s
USER=root
SHLVL=1
DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
MAIL=/var/spool/mail/root
_=/usr/bin/env
OLDPWD=/llama/llama-models-main/models
I've tried searching the internet for answers, but no one seems to have exactly the same issues that I have, or they are running the package on a completely different O.S.
I'd appreciate a little direction here. NOTE, I've already started reading the fine tunning docs, but I'm not sure that is something I want to jump into yet having only started working on meta llama, or any AI for that mater since this Monday. If it will help, I can upload the script I've modified for a review.