Linux command line notes

Questions:

1. i'm the owner of a file, but i cannot delete the file, why? (a tricky problem i encountered before, it is because you must have write permission to the parent directory before you can delete the file)

readlink:

readlink -f file_relative_path #prints the absolute path of a file or symbolic link

#-f the last component of the path must exist

diff:

1. recursively diff files in two direcotries:

diff -r dir0 dir1

2. -q consice output showing only the differences

3. tkdiff is a guide version, not very useful, but still works in some case. (gvim -d is much better in accuracy)

4. diff --suppress-common-lines -y file1 file2 | wc -l (count number of differences)

uniq:

#to reorge and remove duplicate lines for input

%uniq

#to only show the duplicated entries of original input, very useful

%uniq -d

#to only show the uniq lines of the original input

%uniq -u

#easy one to find the duplicated files (same filenames)

%find . -type f -execdir echo {} ';'|sort|uniq -d

cp:

cp --parents sourc destdir #keep full source directory path, warning: the destdir must be an existing directory name

awk:

ask tutorial
awk - delimited output
#-F original delimiter, -v OFS=* is output delimiter

%awk -F":" -v OFS=',' '{print $1, $2, 9}' fname

#substitue, eg. removing white space

%awk '{gsub(/ /, ""); print $2}'

sed (Stream EDitor):

sed 's/word1/word2/g' input.file
sed 's/word1/word2/g' input.file > output.file

#in-place change file
sed -i 's/word1/word2/g' input.file
sed -i -e 's/word1/word2/g' -e 's/xx/yy/g' input.file
## use + separator instead of /
sed -i 's+regex+new-text+g' file.txt

## you can add I option to case insensitive search, only works with GNU sed.
sed -i 's/word1/word2/gI' input
# use ; to concatenate multiple expressions
sed -e 's/word1/word2/g;s/word3/word4/g' #equivalent to below
sed -e 's/word1/word2/g' -e 's/word3/word4/g'
#cool trick in MAC to do equivalent to tree
%alias tree="find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'"

#bulk replacement of string for whole folder

find . -type f| xargs see -i ‘s/pattern_org/pattern_new/g’

xargs:

Linux and Unix xargs command tutorial with examples
Clearly using xargs is far more efficient. In fact several benchmarks suggest using xargs over exec {} is six times more efficient.
#-t will print each command, then immediately execute it:
find . -name "filename" | xargs -t tail

#you can use -I to define a place-holder which will be replaced with each value of the arguments fed to xargs. For example ({} can be changed to any replace_string),

ls -1 | xargs -I '{}' echo '{}'

xargs -I '{}' tail '{}'/'{}'.run.log

find and grep:

FIND -EXEC VS. FIND | XARGS

#!!!find by default wont follow symbolic link, to enable it use "-L", man for help

find xxx -L
#find files but not directory names
find . -type f

#find directories

find . type d
#find files specify depth
find . -maxdepth 2
#TODO: -print option?
#find with inode number
find . -inum <234567>

#follow symbolic link, except when broken

find . -name xxx -L

#find symbolic links

find . -type l

#find but do not print the directory hierarchy

find . [other options] -execdir echo {} ';'

Key mapping issues:

Backspace and Delete key reversed

map Caps Lock to Ctrl key in Gnome

cURL:

curl -O [url] #save to local with the save file name
curl -o [shortname] [url] #save file as another name
curl -o ~/Desktop/localexample.dmg http://url-to-file/example.dmg #save to another directory
curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip #test download speed with curl
curl -O [URL 1] [URL 2] [URL 3] #download multiple files concurrently
curl -L -O -C - url #Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.
-L #Follow location if HTTP 3xx status code found. For example, redirect url. in this case, if -L is ommitted, the actual file will not be downloaded
curl Command Resume Broken DownloadHow to Run Speed Test from the Command Line to Check Internet Connection Speed

# download all jpg files named cat01.jpg to cat20.jpg
curl -O http://example.org/xyz/cat[01-20].jpg

Other useful options are:
--referer http://example.org/ → set a referer (that is, a link you came from)
--user-agent "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" → set user agent, in case the site needs that.


xterm:
#send a terminal to another host
xterm -display <IP>:0
xterm -display <hostname>:0

wget:

# download a file
wget http://example.org/somedir/largeMovie.mov
# download website, 2 levels deep, wait 9 sec per page
wget --wait=9 --recursive --level=2 http://example.org/

Some sites check on user agent. (user agent basically means browser). so you might add this option “--user-agent=”.
wget http://example.org/ --user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36'

wget -r -np -k http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/
use -r (recursive), -np (don't follow links to parent directories), and -k to make links in downloaded HTML or CSS point to local files

Other useful options:
-nd (no directories): download all files to the current directory
-e robots.off: ignore robots.txt files, don't download robots.txt files
-A png,jpg: accept only files with the extensions png or jpg
-m (mirror): -r --timestamping --level inf --no-remove-listing
-nc, --no-clobber: Skip download if files exist

tcsh:

Learn X in Y minutes where X=tcsh

tar:

#to tar
tar -cvf fileName.tar file1 file2 file3
tar -cvf fileName.tar dir1 dir2 dir3
tar -cvf fileName.tar file1 dir1

#to compress
$ tar -cvzf docs.tar.gz /home/vivek/Documents/
$ tar -cvjf docs.tar.bz2 /home/vivek/Documents/

#to list contents of tar
$ tar -tvf docs.tar
$ tar -tvzf docs.tar.gz
$ tar -tvjf docs.tar.bz2

#to extract
$ tar -xvf docs.tar
$ tar -xvzf docs.tar.gz
$ tar -xvjf docs.tar.bz2

rsync:

Linux: Sync Across Machines, rsync
For example, here's what i use to sync/upload my website on my local machine to my server.
rsync -z -a -v -t --exclude="*~"  --exclude=".DS_Store" --exclude=".bash_history"  --exclude="*/_curves_robert_yates/*.png" --exclude="logs/*"  --exclude="xlogs/*" --delete --rsh="ssh -l joe" ~/web/ joe@example.com:~/

-a → archived mode, basically making the file's meta data (owner/perm/timestamp) same as the local file (when possible) and do recursive (i.e. Upload the whole dir).

-P or --progress: this will nicely show the progress of copying
-z → use compression for transmission. (compress files first, transmit, uncompress. This saves bandwidth.)
-v → verbose mode. Print out which files is being updated.
-t → copy timestamp from source to destination. If you don't, rsync will basically update every file. Timestamp is used by rsync to check if file's been updated. -a implies -t.
--exclude=glob_pattern → ignore file names that matches glob_pattern in source directory. (i.e. if it matches, don't upload it, nor delete it on remote server) For example, *.javac means all files ending in .javac
--delete → if a file/dir in destination is not in source directory, delete it.

-n this is cool dry run option, see what will be copied before performing real operation, and if everything is ok, remove -n from command line.

Here's a example of syncing Windows and Mac.
rsync -z -r -v --delete --rsh="ssh -l xah" ~/web/ xah@169.254.125.147:~/web/
Note that -r is used instead of -a. The -r means recursive, all sub directories and files. Don't use -a because that will sync file owner, group, permissions, and others, but because Windows and unix have different permission systems and file systems, so -a is usually not what you want.

Here's a example of reverse direction.
rsync -z -a -v -t --rsh="ssh -l joe" joe@example.org:~/web/ ~/

copy files from one location to another, excluding certain files/folders:
rsync -av --progress sourcefolder /destinationfolder --exclude thefoldertoexclude --exclude anotherfoldertoexclude

Debian/Ubuntu:

#update system packages:

sudo apt-get update

sudo apt-get upgrade

#alternative 

sudo apt-get update && sudo apt-get upgrade -y #-y will provide the confirmation automatically

#search package name: 

apt-cache search keyword

apt-get update

apt-get install

#uninstall packages

%sudo apt purge --auto-remove package0 package1 package2...

sleep:

delay in shell script
%sleep 5s #delay 5 seconds, m: minute, h: hour, d: day.

ps:

%ps auxww #display in bsd format

The aux options are as follows: 

a = show processes for all users 

u = display the process’s user/owner 

x = also show processes not attached to a terminal

%ps -x #process owned by you

timeout:

%timeout 10s command #send soft kill (SIGTERM) in 10 seconds, this may be ignored by the command
%timeout -s SIGKILL 10s command #force kill (SIGKILL) in 10 seconds, cannot be ignored by command
%timeout -k 10 5 command #send soft kill in 5s, if command not terminated, then force kill it 5s after that(10s=5+5).

Perforce:

#compare differences between two labels:
%p4    diff2    -q    //...@label_name_0    //...@label_name_1

ls:

#list inode of file
ls -i

git:

git clone https://github.com/github.com/libopencm3/libopencm3.git

git clone xxx new_dir_name

system task:

#show system info in ASCII format

screenfetch

#change password

sudo passwd root

sudo passwd account_name

#add a new user

sudo adduser user_name #ubuntu/debian

sudo useradd -m newuser #arch/manjaro

#add user to sudo group

sudo usermod -aG sudo username #debian/ubuntu

sudo usermod -aG wheel <username> #arch/manjaro

#test

su - username

#delete user

sudo deluser --remove-home newuser #debian/ubuntu

sudo userdel -r newuser #arch/manjaro

#shutdown system

%shutdown

#restart linux

%shutdown -r

#shutdown or restart immediately

%shutdown now

%shudown -r now

#scheduled shutdown

%shutdown 20          #shutdown in 20 mins

%shutdown -h 17:30 #shutdown at 17:30

%shutdown -r 17:30  #restart at 17:30

#cancel scheduled shutdown

%shutdown -c

#add shutdown wall message

%shutdown 'YOUR WALL MESSAGE'

Mount disk:

%lsblk      #show the partition tables of all disks, note if there is partion on your disk, you need to mount the partition name instead of the disk name

%dmesg | tail    #after connect the usb disk, run this command to find the name of the disk. normalled in /dev/sd*

%sudo mount /dev/sdb1 /mnt/pico

%umount device|directory. #safely unjoint device before you unplug it

mount error: wrong fs type, bad option, bad superblock

List Devices:

%lspci      #list pcie devices

#example to list macbook pcie devices

%lspci | grep -E  'Atheros|Broadcom' 

%lsusb

Enable SSH remote connection:

%sudo apt install openssh-server

%sudo systemctl start ssh

%sudo systemctl enable ssh

%sudo systemctl status ssh #check current ssh status

Change default boot into Command line or GUI:

online tutorial

%systemctl get-default

%sudo systemctl set-default multi-user.target

%sudo systemctl reboot

#to switch back to gui, replace above command correspondingly

%sudo systemctl set-default graphical.target

Command history:

%cmd<Ctrl-r>     #search revsersely through command history and auto complete

%<Esc> #put command on screen(<Enter> will execute that command

%<Ctrl-E> #similar to <Esc>

Command line settings:

#configure the looks of non gui mode command line:

sudo dpkg-reconfigure console-setup

#set time zone

timedatectl list-timezones

sudo timedatectl set-timezone "America/Los_Angeles"

tmux:

prefix key Ctrl-b

%tmux ls  # list all open tmux sessions

%Ctrl-b d #detach session

%tmux attach -t session-name #attach to a existing session

%tmux new-s session-name #create new session

%tmux rename-session -t session-old-name session-new-name

%tmux kill-session -t session-name

#inside tmux

%Ctrl-b % #create new vertical pane

%Ctrl-b " #create new horizontal pane

%Ctrl-b window-number #switch to a selected window

%Ctrl-b c #create a new window

%Ctrl-b Arrow-key #switch to pane

Battery Status

%upower -e #same as —enumerate, enumerates object paths for devices

%upower -i device path #display information about the laptop battery (replace the path below with your battery's path