Shell scripts to count files and sum up file sizes

Compress sparse files preserving their holes

use -S option for tar

tar -Sczvf sparse_file.tar.gz sparse_file

Find FIFO (named pipe) in Linux

 find ./ -type p 

Find sparse files in Linux

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print}'

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}'

To find and remove the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs rm -f

To find and compress the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs -I {} sh -c "tar -Sczvf {}.tar.gz {}; rm -f {}"

My zsh prompt setup

Add following line to ~/.zshrc:

PROMPT='%F{red}%*%f %F{blue}%1~%f %# '

More details about ZSH prompt: https://zsh.sourceforge.io/Doc/Release/Prompt-Expansion.html

Bash: print leading zeros for numbers in a for loop

• for ((i=1; i>=100; i++)); do
      echo $(printf "%03d" $i)
  done

• BASH 4+

  for i in {001..100}; do
      echo "$i"
  done

  create 100 100k files:

  for i in {001..100}; do dd if=/dev/urandom of=100K_${i}.dat  bs=1024  count=100; done

Use grep to extract substring between two specific characters/words

Input:

00024D9E: (0051,100F) LO 4 [A11]

Output:

A11

To achieve the above,

echo "00024D9E: (0051,100F) LO 4 [A11]" | grep -o -P '(?<=\[).*(?=\])'

Bash - print number with leading zeros

# 00001
echo $(printf '%05d' 1)


# 00123
n=123
echo $(printf '%05d' $n) 

Find files and set their permissions

find files and pipe them to chmod to set permissions:

find . -type d -exec chmod 0755 {} \;
find . -type f -exec chmod 0644 {} \;

Bash: check if two paths refer to the same file

file1 -ef file2
True if file1 and file2 refer to the same device and inode numbers.

if [[ "$HOME/Downloads/file1" -ef "Downloads/file1" ]]; then
    echo "Same file"
fi

see also

• Bash conditional expressions

sed replace string in a file contains slash

sed -i "" "s|AAA/BBB|AAA/CCC|g" input-file.txt

or

sed -i ".bak" "s|AAA/BBB|AAA/CCC|g" input-file.txt

Note:

• We want in place update to the file, so we need --in-place or -i option, and on Mac OS, the in place extension/suffix must be supplied even if it is a empty string: -i ""
• We cannot use / as the delimiter

Bash: check if string contains substring

• method 1:

  if [[ $str == *"AAA"* ]]; then
      echo "yes"
  else
  	echo "no"
  fi
  	

• method 2:

  if [[ $str =~ "AAA" ]]; then
      echo "yes"
  else
  	echo "no"
  fi
  	

see also

• How to check if a string contains a substring in Bash

Build Java runtime from OpenJDK using jlink

MODULES=$(java --list-modules | sed 's/\@.*//' | paste -sd "," -)
jlink --no-header-files --no-man-pages --compress=2 --add-modules $MODULES --output java-runtime

Bash: join lines with specific delimiter char

ls -1 | paste -sd "," -

BASH check os type

if [[ "$OSTYPE" == "linux-gnu" ]]; then
    echo linux
elif [[ "$OSTYPE" == "darwin"* ]]; then
    echo mac_os
elif [[ "$OSTYPE" == "cygwin" ]]; then
    echo windows
elif [[ "$OSTYPE" == "msys" ]]; then
    echo windows
elif [[ "$OSTYPE" == "win32" ]]; then
    echo windows
elif [[ "$OSTYPE" == "freebsd"* ]]; then
    echo freebsd
else
    echo unknown
fi

Shell scripting: Sum file sizes

Linux

• find . -type f -printf %s\\n | paste -sd+ | bc

• find . -type f -printf %s\\n | numsum

Mac OS (BSD)

• find . -type f -print0 | xargs -0 stat -f '%z ' | paste -sd+ - | bc

Sum numbers from lines of file

paste -s -d + file.txt | bc

Shell script: loop through find results

find . -type f |while read file 
do
    size=$(stat --printf=%s $file)
    csum=$(cksum ${file} | awk '{print $1}')
    echo "$file $size $csum"
done

One line shell script: generate large number of dummy files

The following command generate 10,000 dummy files (16KB)

i=0; while [[ $i -lt 10000 ]]; do dd if=/dev/urandom bs=1024 count=16 of=dummy${i}; i=$((i+1)); done

On Mac OS:

i=0; while [[ $i -lt 10000 ]]; do mkfile 16k dummy${i}; i=$((i+1)); done

Get the total size of files in a directory

find . -type f -print0 | xargs -0 stat --format=%s | gawk '{s+=$1} END {print s}'

or

find . -type f -print0 | xargs -0 stat --format=%s | awk '{s+=$1} END {print s}'