Compress sparse files preserving their holes

use -S option for tar

tar -Sczvf sparse_file.tar.gz sparse_file

Find FIFO (named pipe) in Linux

 find ./ -type p 

Find sparse files in Linux

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print}'

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}'

To find and remove the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs rm -f

To find and compress the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs -I {} sh -c "tar -Sczvf {}.tar.gz {}; rm -f {}"

Bash: print leading zeros for numbers in a for loop

• for ((i=1; i>=100; i++)); do
      echo $(printf "%03d" $i)
  done

• BASH 4+

  for i in {001..100}; do
      echo "$i"
  done

  create 100 100k files:

  for i in {001..100}; do dd if=/dev/urandom of=100K_${i}.dat  bs=1024  count=100; done

Configure Git-Bash for Windows Terminal

• Open Settings tab with Ctrl+,, the click Gear icon to Open settings.json file
• Add/Edit Git Bash profile:

  {
      "guid": "{2ece5bfe-50ed-5f3a-ab87-5cd4baafed2b}",
      "hidden": false,
      "name": "Git Bash",
      "source": "Git",
      "fontFace" : "Consolas",
      "fontSize" : 11,
      "startingDirectory" : "%USERPROFILE%",
      "acrylicOpacity" : 0.75,
      "closeOnExit" : true,
      "colorScheme" : "Campbell",
      "cursorColor" : "#FFFFFF",
      "cursorShape" : "bar",
      "historySize" : 9001,
      "padding" : "0, 0, 0, 0",
      "snapOnInput" : true,
      "useAcrylic" : true
  }

see also

• Adding Git-Bash to the new Windows Terminal

Use grep to extract substring between two specific characters/words

Input:

00024D9E: (0051,100F) LO 4 [A11]

Output:

A11

To achieve the above,

echo "00024D9E: (0051,100F) LO 4 [A11]" | grep -o -P '(?<=\[).*(?=\])'

Count lines of Java source code

find . -type f -name "*.java" | xargs cat | grep [alnum{}] | wc -l

or shell script could be:

#!/bin/bash

[[ -z $1 ]] && dir=. || dir=$1

find $dir -type f -name "*.java" | xargs cat | grep [alnum{}] | wc -l

Bash: check if two paths refer to the same file

file1 -ef file2
True if file1 and file2 refer to the same device and inode numbers.

if [[ "$HOME/Downloads/file1" -ef "Downloads/file1" ]]; then
    echo "Same file"
fi

see also

• Bash conditional expressions

Bash: replace substring using regular expressions

With bash, you can replace string using ${variable//pattern/replacement}

a="<token>XXX</token>"

# to replace the token element value with YYY
a=${a//<token>[a-zA-Z0-9]*<\/token>/<token>[a-zA-Z0-9]*<\/token>}

see also

• Search and replace in bash using regular expressions

sed: replace string between two matching patterns

# settings.xml
# <settings><token>xxx</token></settings>

cat settings.xml | sed -e "s|<token>.*</token>|<token>yyy</token>|g"

# <settings><token>yyy</token></settings>

see also

• replace a unknown string between two known strings with sed

sed replace string in a file contains slash

sed -i "" "s|AAA/BBB|AAA/CCC|g" input-file.txt

or

sed -i ".bak" "s|AAA/BBB|AAA/CCC|g" input-file.txt

Note:

• We want in place update to the file, so we need --in-place or -i option, and on Mac OS, the in place extension/suffix must be supplied even if it is a empty string: -i ""
• We cannot use / as the delimiter

Bash: check if string contains substring

• method 1:

  if [[ $str == *"AAA"* ]]; then
      echo "yes"
  else
  	echo "no"
  fi
  	

• method 2:

  if [[ $str =~ "AAA" ]]; then
      echo "yes"
  else
  	echo "no"
  fi
  	

see also

• How to check if a string contains a substring in Bash

Build Java runtime from OpenJDK using jlink

MODULES=$(java --list-modules | sed 's/\@.*//' | paste -sd "," -)
jlink --no-header-files --no-man-pages --compress=2 --add-modules $MODULES --output java-runtime

Bash: join lines with specific delimiter char

ls -1 | paste -sd "," -

sed: delete from matching pattern to the end of the line

sed 's/\.com\/.*/.com/' file.txt

example 2

Command:

java --list-modules

Result:

java.base@13.0.1
java.compiler@13.0.1
java.datatransfer@13.0.1
java.desktop@13.0.1
java.instrument@13.0.1
java.logging@13.0.1
java.management@13.0.1
java.management.rmi@13.0.1
java.naming@13.0.1
java.net.http@13.0.1
java.prefs@13.0.1
java.rmi@13.0.1
java.scripting@13.0.1
java.se@13.0.1
java.security.jgss@13.0.1
java.security.sasl@13.0.1
java.smartcardio@13.0.1
java.sql@13.0.1
java.sql.rowset@13.0.1
java.transaction.xa@13.0.1
java.xml@13.0.1
java.xml.crypto@13.0.1
jdk.accessibility@13.0.1
jdk.aot@13.0.1
jdk.attach@13.0.1
jdk.charsets@13.0.1
jdk.compiler@13.0.1
jdk.crypto.cryptoki@13.0.1
jdk.crypto.ec@13.0.1
jdk.dynalink@13.0.1
jdk.editpad@13.0.1
jdk.hotspot.agent@13.0.1
jdk.httpserver@13.0.1
jdk.internal.ed@13.0.1
jdk.internal.jvmstat@13.0.1
jdk.internal.le@13.0.1
jdk.internal.opt@13.0.1
jdk.internal.vm.ci@13.0.1
jdk.internal.vm.compiler@13.0.1
jdk.internal.vm.compiler.management@13.0.1
jdk.jartool@13.0.1
jdk.javadoc@13.0.1
jdk.jcmd@13.0.1
jdk.jconsole@13.0.1
jdk.jdeps@13.0.1
jdk.jdi@13.0.1
jdk.jdwp.agent@13.0.1
jdk.jfr@13.0.1
jdk.jlink@13.0.1
jdk.jshell@13.0.1
jdk.jsobject@13.0.1
jdk.jstatd@13.0.1
jdk.localedata@13.0.1
jdk.management@13.0.1
jdk.management.agent@13.0.1
jdk.management.jfr@13.0.1
jdk.naming.dns@13.0.1
jdk.naming.rmi@13.0.1
jdk.net@13.0.1
jdk.pack@13.0.1
jdk.rmic@13.0.1
jdk.scripting.nashorn@13.0.1
jdk.scripting.nashorn.shell@13.0.1
jdk.sctp@13.0.1
jdk.security.auth@13.0.1
jdk.security.jgss@13.0.1
jdk.unsupported@13.0.1
jdk.unsupported.desktop@13.0.1
jdk.xml.dom@13.0.1
jdk.zipfs@13.0.1

Command:

java --list-modules | sed 's/\@.*//'

Result:

java.base
java.compiler
java.datatransfer
java.desktop
java.instrument
java.logging
java.management
java.management.rmi
java.naming
java.net.http
java.prefs
java.rmi
java.scripting
java.se
java.security.jgss
java.security.sasl
java.smartcardio
java.sql
java.sql.rowset
java.transaction.xa
java.xml
java.xml.crypto
jdk.accessibility
jdk.aot
jdk.attach
jdk.charsets
jdk.compiler
jdk.crypto.cryptoki
jdk.crypto.ec
jdk.dynalink
jdk.editpad
jdk.hotspot.agent
jdk.httpserver
jdk.internal.ed
jdk.internal.jvmstat
jdk.internal.le
jdk.internal.opt
jdk.internal.vm.ci
jdk.internal.vm.compiler
jdk.internal.vm.compiler.management
jdk.jartool
jdk.javadoc
jdk.jcmd
jdk.jconsole
jdk.jdeps
jdk.jdi
jdk.jdwp.agent
jdk.jfr
jdk.jlink
jdk.jshell
jdk.jsobject
jdk.jstatd
jdk.localedata
jdk.management
jdk.management.agent
jdk.management.jfr
jdk.naming.dns
jdk.naming.rmi
jdk.net
jdk.pack
jdk.rmic
jdk.scripting.nashorn
jdk.scripting.nashorn.shell
jdk.sctp
jdk.security.auth
jdk.security.jgss
jdk.unsupported
jdk.unsupported.desktop
jdk.xml.dom
jdk.zipfs

see also

• How to delete the rest of each line after a certain pattern or a string in a file?

Bash: generate files from random data

• create 10 x 1KB files

  for i in {1..10}; do dd if=/dev/urandom bs=1k count=1 of=1KB.$i; done
  

• create 10 x 1MB files

  for i in {1..10}; do dd if=/dev/urandom bs=1m count=1 of=1MB.$i; done
  

• create 10 x 1GB files

  for i in {1..10}; do dd if=/dev/urandom bs=1m count=1000 of=1GB.$i; done
  

BASH check os type

if [[ "$OSTYPE" == "linux-gnu" ]]; then
    echo linux
elif [[ "$OSTYPE" == "darwin"* ]]; then
    echo mac_os
elif [[ "$OSTYPE" == "cygwin" ]]; then
    echo windows
elif [[ "$OSTYPE" == "msys" ]]; then
    echo windows
elif [[ "$OSTYPE" == "win32" ]]; then
    echo windows
elif [[ "$OSTYPE" == "freebsd"* ]]; then
    echo freebsd
else
    echo unknown
fi

Shell scripting: Sum file sizes

Linux

• find . -type f -printf %s\\n | paste -sd+ | bc

• find . -type f -printf %s\\n | numsum

Mac OS (BSD)

• find . -type f -print0 | xargs -0 stat -f '%z ' | paste -sd+ - | bc

Sum numbers from lines of file

paste -s -d + file.txt | bc

Shell script: loop through find results

find . -type f |while read file 
do
    size=$(stat --printf=%s $file)
    csum=$(cksum ${file} | awk '{print $1}')
    echo "$file $size $csum"
done