Linux command line notes

Questions:

1. i'm the owner of a file, but i cannot delete the file, why? (a tricky problem i encountered before, it is because you must have write permission to the parent directory before you can delete the file)

readlink:

readlink -f file_relative_path #prints the absolute path of a file or symbolic link

#-f the last component of the path must exist

diff:

1. recursively diff files in two direcotries:

diff -r dir0 dir1

2. -q consice output showing only the differences

3. tkdiff is a guide version, not very useful, but still works in some case. (gvim -d is much better in accuracy)

4. diff --suppress-common-lines -y file1 file2 | wc -l (count number of differences)

uniq:

#to reorge and remove duplicate lines for input

%uniq

#to only show the duplicated entries of original input, very useful

%uniq -d

#to only show the uniq lines of the original input

%uniq -u

#easy one to find the duplicated files (same filenames)

%find . -type f -execdir echo {} ';'|sort|uniq -d

cp:

cp --parents sourc destdir #keep full source directory path, warning: the destdir must be an existing directory name

awk:

ask tutorial
awk - delimited output
#-F original delimiter, -v OFS=* is output delimiter

%awk -F":" -v OFS=',' '{print $1, $2, 9}' fname

#substitue, eg. removing white space

%awk '{gsub(/ /, ""); print $2}'

sed (Stream EDitor):

sed 's/word1/word2/g' input.file
sed 's/word1/word2/g' input.file > output.file

#in-place change file
sed -i 's/word1/word2/g' input.file
sed -i -e 's/word1/word2/g' -e 's/xx/yy/g' input.file
## use + separator instead of /
sed -i 's+regex+new-text+g' file.txt

## you can add I option to case insensitive search, only works with GNU sed.
sed -i 's/word1/word2/gI' input
# use ; to concatenate multiple expressions
sed -e 's/word1/word2/g;s/word3/word4/g' #equivalent to below
sed -e 's/word1/word2/g' -e 's/word3/word4/g'
#cool trick in MAC to do equivalent to tree
%alias tree="find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'"

#bulk replacement of string for whole folder

find . -type f| xargs see -i ‘s/pattern_org/pattern_new/g’

xargs:

Linux and Unix xargs command tutorial with examples
Clearly using xargs is far more efficient. In fact several benchmarks suggest using xargs over exec {} is six times more efficient.
#-t will print each command, then immediately execute it:
find . -name "filename" | xargs -t tail

#you can use -I to define a place-holder which will be replaced with each value of the arguments fed to xargs. For example ({} can be changed to any replace_string),

ls -1 | xargs -I '{}' echo '{}'

xargs -I '{}' tail '{}'/'{}'.run.log

find and grep:

FIND -EXEC VS. FIND | XARGS

#!!!find by default wont follow symbolic link, to enable it use "-L", man for help

find xxx -L
#find files but not directory names
find . -type f

#find directories

find . type d
#find files specify depth
find . -maxdepth 2
#TODO: -print option?
#find with inode number
find . -inum <234567>

#follow symbolic link, except when broken

find . -name xxx -L

#find symbolic links

find . -type l

#find but do not print the directory hierarchy

find . [other options] -execdir echo {} ';'

Key mapping issues:

Backspace and Delete key reversed

map Caps Lock to Ctrl key in Gnome

cURL:

curl -O [url] #save to local with the save file name
curl -o [shortname] [url] #save file as another name
curl -o ~/Desktop/localexample.dmg http://url-to-file/example.dmg #save to another directory
curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip #test download speed with curl
curl -O [URL 1] [URL 2] [URL 3] #download multiple files concurrently
curl -L -O -C - url #Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.
-L #Follow location if HTTP 3xx status code found. For example, redirect url. in this case, if -L is ommitted, the actual file will not be downloaded
curl Command Resume Broken DownloadHow to Run Speed Test from the Command Line to Check Internet Connection Speed

# download all jpg files named cat01.jpg to cat20.jpg
curl -O http://example.org/xyz/cat[01-20].jpg

Other useful options are:
--referer http://example.org/ → set a referer (that is, a link you came from)
--user-agent "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" → set user agent, in case the site needs that.


xterm:
#send a terminal to another host
xterm -display <IP>:0
xterm -display <hostname>:0

wget:

# download a file
wget http://example.org/somedir/largeMovie.mov
# download website, 2 levels deep, wait 9 sec per page
wget --wait=9 --recursive --level=2 http://example.org/

Some sites check on user agent. (user agent basically means browser). so you might add this option “--user-agent=”.
wget http://example.org/ --user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36'

wget -r -np -k http://www.ime.usp.br/~coelho/mac0122-2013/ep2/esqueleto/
use -r (recursive), -np (don't follow links to parent directories), and -k to make links in downloaded HTML or CSS point to local files

Other useful options:
-nd (no directories): download all files to the current directory
-e robots.off: ignore robots.txt files, don't download robots.txt files
-A png,jpg: accept only files with the extensions png or jpg
-m (mirror): -r --timestamping --level inf --no-remove-listing
-nc, --no-clobber: Skip download if files exist

tcsh:

Learn X in Y minutes where X=tcsh

tar:

#to tar
tar -cvf fileName.tar file1 file2 file3
tar -cvf fileName.tar dir1 dir2 dir3
tar -cvf fileName.tar file1 dir1

#to compress
$ tar -cvzf docs.tar.gz /home/vivek/Documents/
$ tar -cvjf docs.tar.bz2 /home/vivek/Documents/

#to list contents of tar
$ tar -tvf docs.tar
$ tar -tvzf docs.tar.gz
$ tar -tvjf docs.tar.bz2

#to extract
$ tar -xvf docs.tar
$ tar -xvzf docs.tar.gz
$ tar -xvjf docs.tar.bz2

rsync:

Linux: Sync Across Machines, rsync
For example, here's what i use to sync/upload my website on my local machine to my server.
rsync -z -a -v -t --exclude="*~"  --exclude=".DS_Store" --exclude=".bash_history"  --exclude="*/_curves_robert_yates/*.png" --exclude="logs/*"  --exclude="xlogs/*" --delete --rsh="ssh -l joe" ~/web/ joe@example.com:~/

-a → archived mode, basically making the file's meta data (owner/perm/timestamp) same as the local file (when possible) and do recursive (i.e. Upload the whole dir).

-P or --progress: this will nicely show the progress of copying
-z → use compression for transmission. (compress files first, transmit, uncompress. This saves bandwidth.)
-v → verbose mode. Print out which files is being updated.
-t → copy timestamp from source to destination. If you don't, rsync will basically update every file. Timestamp is used by rsync to check if file's been updated. -a implies -t.
--exclude=glob_pattern → ignore file names that matches glob_pattern in source directory. (i.e. if it matches, don't upload it, nor delete it on remote server) For example, *.javac means all files ending in .javac
--delete → if a file/dir in destination is not in source directory, delete it.

-n this is cool dry run option, see what will be copied before performing real operation, and if everything is ok, remove -n from command line.

Here's a example of syncing Windows and Mac.
rsync -z -r -v --delete --rsh="ssh -l xah" ~/web/ xah@169.254.125.147:~/web/
Note that -r is used instead of -a. The -r means recursive, all sub directories and files. Don't use -a because that will sync file owner, group, permissions, and others, but because Windows and unix have different permission systems and file systems, so -a is usually not what you want.

Here's a example of reverse direction.
rsync -z -a -v -t --rsh="ssh -l joe" joe@example.org:~/web/ ~/

copy files from one location to another, excluding certain files/folders:
rsync -av --progress sourcefolder /destinationfolder --exclude thefoldertoexclude --exclude anotherfoldertoexclude

Debian/Ubuntu:

#update system packages:

sudo apt-get update

sudo apt-get upgrade

#alternative 

sudo apt-get update && sudo apt-get upgrade -y #-y will provide the confirmation automatically

#search package name: 

apt-cache search keyword

apt-get update

apt-get install

#uninstall packages

%sudo apt purge --auto-remove package0 package1 package2...

sleep:

delay in shell script
%sleep 5s #delay 5 seconds, m: minute, h: hour, d: day.

ps:

%ps auxww #display in bsd format

The aux options are as follows: 

a = show processes for all users 

u = display the process’s user/owner 

x = also show processes not attached to a terminal

%ps -x #process owned by you

timeout:

%timeout 10s command #send soft kill (SIGTERM) in 10 seconds, this may be ignored by the command
%timeout -s SIGKILL 10s command #force kill (SIGKILL) in 10 seconds, cannot be ignored by command
%timeout -k 10 5 command #send soft kill in 5s, if command not terminated, then force kill it 5s after that(10s=5+5).

Perforce:

#compare differences between two labels:
%p4    diff2    -q    //...@label_name_0    //...@label_name_1

ls:

#list inode of file
ls -i

git:

git clone https://github.com/github.com/libopencm3/libopencm3.git

git clone xxx new_dir_name

system task:

#show system info in ASCII format

screenfetch

#change password

sudo passwd root

sudo passwd account_name

#add a new user

sudo adduser user_name #ubuntu/debian

sudo useradd -m newuser #arch/manjaro

#add user to sudo group

sudo usermod -aG sudo username #debian/ubuntu

sudo usermod -aG wheel <username> #arch/manjaro

#test

su - username

#delete user

sudo deluser --remove-home newuser #debian/ubuntu

sudo userdel -r newuser #arch/manjaro

#shutdown system

%shutdown

#restart linux

%shutdown -r

#shutdown or restart immediately

%shutdown now

%shudown -r now

#scheduled shutdown

%shutdown 20          #shutdown in 20 mins

%shutdown -h 17:30 #shutdown at 17:30

%shutdown -r 17:30  #restart at 17:30

#cancel scheduled shutdown

%shutdown -c

#add shutdown wall message

%shutdown 'YOUR WALL MESSAGE'

Mount disk:

%lsblk      #show the partition tables of all disks, note if there is partion on your disk, you need to mount the partition name instead of the disk name

%dmesg | tail    #after connect the usb disk, run this command to find the name of the disk. normalled in /dev/sd*

%sudo mount /dev/sdb1 /mnt/pico

%umount device|directory. #safely unjoint device before you unplug it

mount error: wrong fs type, bad option, bad superblock

List Devices:

%lspci      #list pcie devices

#example to list macbook pcie devices

%lspci | grep -E  'Atheros|Broadcom' 

%lsusb

Enable SSH remote connection:

%sudo apt install openssh-server

%sudo systemctl start ssh

%sudo systemctl enable ssh

%sudo systemctl status ssh #check current ssh status

Change default boot into Command line or GUI:

online tutorial

%systemctl get-default

%sudo systemctl set-default multi-user.target

%sudo systemctl reboot

#to switch back to gui, replace above command correspondingly

%sudo systemctl set-default graphical.target

Command history:

%cmd<Ctrl-r>     #search revsersely through command history and auto complete

%<Esc> #put command on screen(<Enter> will execute that command

%<Ctrl-E> #similar to <Esc>

Command line settings:

#configure the looks of non gui mode command line:

sudo dpkg-reconfigure console-setup

#set time zone

timedatectl list-timezones

sudo timedatectl set-timezone "America/Los_Angeles"

tmux:

prefix key Ctrl-b

%tmux ls  # list all open tmux sessions

%Ctrl-b d #detach session

%tmux attach -t session-name #attach to a existing session

%tmux new-s session-name #create new session

%tmux rename-session -t session-old-name session-new-name

%tmux kill-session -t session-name

#inside tmux

%Ctrl-b % #create new vertical pane

%Ctrl-b " #create new horizontal pane

%Ctrl-b window-number #switch to a selected window

%Ctrl-b c #create a new window

%Ctrl-b Arrow-key #switch to pane

Battery Status

%upower -e #same as —enumerate, enumerates object paths for devices

%upower -i device path #display information about the laptop battery (replace the path below with your battery's path

Systemverilog Notes

• Systemverilog simulation steps

• asdfasd

• Gotcha: The Behavior of Foreach Loop Variables Depends on How the Array Dimensions Are Specified
• How is for-loop synthesizable in any HDL (Verilog)? How is it implemented in hardware?
• Solvnet Synopsys ETHERNET SVT UVM main page (also read the FAQ pdf, it's very detailed)
• Solvnet Synopsys Ethernet VIP Interface Reference
• Systemverilog process class: The  process::self() function returns a handle to the current process, that is, a handle to the process making the call. SystemVerilog: Fine grain process control
• the repeat loop, expression is evaluated only once at the beginning, other loops, expression is re-evaluated in every iteration: loop evaluation, nandland for loop
• verilog generate: nandland generate example
• Virtual Class and upcasting
• Systemverilog-abstract-classes/
  
• data types:
• wire (a type of net): 

• A net represent connections between hardware elements. Just as in real circuits, nets have values continuously driven on them by the outputs of devices that they are connected to.
• can only be driven with a continuous assignment statement
• a net is the only construct that resolves the effect of different states and strengths simultaneously driving the same signal.
• the behavior of a net is defined by a built-in resolution function using the values and strengths of all the drivers on a net. Every time there is a change on one of the drivers, the function is called to produce a resolved value. The function is create at elaboration (before simulation starts) and is based on the kind of net type, wand, wor, tril, etc.

• logic: 

• can be driven by continuous assignments, gates, and modules, in addition to being a variables.
• can be used anywhere a net is used, except that a logic variable cannot be driven by multiple structural drivers (such as when you are modeling a bidirectional buss)

• continuous assignment, procedural assignment, and classes

• class based testbenches cannot have continuous assignments because classes are dynamically created objects and are not allowed to have structural constructs like continuous assignments. 
• Although a class can read the resolved value of nets, it can only make procedural assignments to variables. Therefore, the testbench needs to create a variable that is continuously assigned to a wire (if you want to have multiple drives to that wire).
• procedural assignments to variables use the simple rule: last write wins. You are not allowed to make procedural assignemtns to nets because there is no way to represent how the value you assigning should be resolved with the other drivers.

• assign/deassign, force/release

• Another form of procedural continuous assignment is provided by the  force and  release procedural statements. These statements have a similar effect to the  assign - deassign pair, but a force can be applied to nets as well as to variables. 
• A  force statement to a variable shall override a procedural assignment, continuous assignment or an assign procedural continuous assignment to the variable until a  release procedural statement is executed on the variable. 
• A  force procedural statement on a net shall override all drivers of the net—gate outputs, module outputs, and continuous assignments—until a  release procedural statement is executed on the net. When released, the net shall immediately be assigned the value determined by the drivers of the net.

• logic vs wire in an interface 

• if your testbench drives an asynchronous signal in an interface with a procedural assignment, the signal must be a logic type. Signals in a clocking block are always synchronous and can be declared as logic or wire(?).
• wire can resolve multiple structural drivers, but logic cannot. choose depending on your user scenarios.

• procedures and procedural assignment

• initial_construct ::=  initial statement_or_null
• always_construct ::= always_keyword statement
• always_keyword ::=  always |  always_comb |  always_latch |  always_ff
• final_construct ::=  final function_statement
• function_declaration ::=  function [ lifetime ] function_body_declaration
• task_declaration ::=  task [ lifetime ] task_body_declaration
• In addition to these structured procedures, SystemVerilog contains other procedural contexts, such as coverage point expressions, assertion sequence match items, and action blocks.

• define and parameters for constant
• Assertions
• Dulos SV Assertion Tutorial
  
• Modules and Hierarchy
• Bind
• Binding is like secretly instantiating a module/interface within another RTL file without disturbing the existing code. The binded module/interface is instantiated directly into the target module. How to bind inner signals from DUT?
• OOP
• singleton classes
• The singleton pattern is implemented by creating a class wit a method that creates a new instance of the class if one does not exist. If an instance already exists, it simply returns a handle to that object. To make sure that he object cannot be instantiated any other way, you must make the constructor protected. Don't make it local, because an extend class might need to access the constructor.
• Number
• Rules for expression types (from LRM). The following are the rules for determining the resulting type of an expression:
• — Expression type depends only on the operands. It does not depend on the left-hand side (if any).
• — Decimal numbers are signed.
• — Based numbers are unsigned, except where the s notation is used in the base specifier (as in 4'sd12 ).
• — Bit-select results are unsigned, regardless of the operands.
• — Part-select results are unsigned, regardless of the operands even if the part-select specifies the entire vector.
• logic [15:0] a;
• logic signed [7:0] b;
• initial
• a = b[7:0]; // b[7:0] is unsigned and therefore zero-extended
• — Concatenate results are unsigned, regardless of the operands.
• — Comparison and reduction operator results are unsigned, regardless of the operands.
• — Reals converted to integers by type coercion are signed
• — The sign and size of any self-determined operand are determined by the operand itself and independent of the remainder of the expression.
• — For non-self-determined operands, the following rules apply:
• — If any operand is real, the result is real.
• — If any operand is unsigned, the result is unsigned, regardless of the operator.
• — If all operands are signed, the result will be signed, regardless of operator, except when specified otherwise.