Find FIFO (named pipe) in Linux

 find ./ -type p 

Find sparse files in Linux

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print}'

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}'

To find and remove the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs rm -f

To find and compress the sparse files:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs -I {} sh -c "tar -Sczvf {}.tar.gz {}; rm -f {}"

SSH public key authentication rejected with debug message: no mutual signature algorithm

ssh user@remote-server -vvvv

debug1: no mutual signature algorithm

Cause

The RSA SHA-1 hash algorithm is being quickly deprecated across operating systems and SSH clients because of various security vulnerabilities, with many of these technologies now outright denying the use of this algorithm.

Solution

Generate SSH key using ED25519 algorithm:

ssh-keygen -t ed25519

See also

• SSH-RSA key rejected with message "no mutual signature algorithm"
• Debugging SSH public key authentication problems

Mac OS: fix duplicated Open With entries

/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -kill -r -domain local -domain user;killall Finder;echo "Open With has been rebuilt, Finder will relaunch"

see also

• Remove Duplicates from the “Open With” Right-Click Menu in Mac OS X

Use GNU grep to count the length of substring between two words

Note: the command below requires GNU grep, it is available on Linux by default. On Mac OS, you need to install it manually using

brew install grep

and command is called ggrep instead of grep.

Get the substring between two words

echo "<person><name>abc</name></person>" | grep -o -P '(?<=<name>).*(?=</name>)'

Output should be:

abc

Count the length of substring

echo "<person><name>abc</name></person>" | grep -o -P '(?<=<name>).*(?=</name>)' | wc -c

Output should be:

3

Count substring lengths for each line

cat multi-lines.txt | while read line; do echo $line | grep -o -P '(?<=<name>).*(?=</name>)' | wc -c; done

see also

• How to use sed/grep to extract text between two words?
• How to install and use GNU Grep in macOS

Use grep to count number of occurs of a specific element in a single line XML text

For example, I can input.txt below, three lines of XML text, I want to count the number of id elements for each line:

<result><id>1</id><id>5</id><id>7</id></result>
<result><id>71</id></result>
<result><id>15</id><id>33</id></result>

I use grep -o:

cat input.txt | while read line; do echo $line | grep -o '<id>' | wc -l; done

The output should be:

3
1
2

MacOS: prevent Spotlight from indexing external drives

To prevent Spotlight from scanning and indexing your external drive, create a file named .metadata_never_index or .metadata_never_index_unless_rootfs at the root of the external drive:

sudo touch /Volumes/ExternalDrive/.metadata_never_index

or

sudo touch /Volumes/ExternalDrive/.metadata_never_index_unless_rootfs

AND OR NOT operators for grep

grep OR

• use '\|'

  grep 'pattern1\|pattern2' file

• use -E

  grep -E 'pattern1|pattern2' file

• use -e

  grep -e pattern1 -e pattern2 file

grep AND

• use pattern1.*pattern2

  grep -E 'pattern1.*pattern2|pattern2.*pattern1' file

• use multiple (piped) grep commands

  grep pattern1 file | grep pattern2

grep NOT (invert) option

• use -v option

  grep -v pattern file

see also

• 7 Linux Grep OR, Grep AND, Grep NOT Operator Examples

Remove CTRL-M characters from a file in Linux or MacOS

You may need to do this when you import a text file from MS-DOS (or MS-Windows), and forget to transfer it in ASCII or text mode. Here are several ways to do it; pick the one you are most comfortable with.

• The easiest way is probably to use sed to remove the ^M characters. Type this command:

  % sed -e "s/^M//" filename > newfilename

  To enter ^M, type CTRL-V, then CTRL-M. That is, hold down the CTRL key then press V and M in succession.
• You can also do it in vi: % vi filename
  Inside vi [in ESC mode] type: :%s/^M//g
  To enter ^M, type CTRL-V, then CTRL-M. That is, hold down the CTRL key then press V and M in succession.
• You can also do it inside Emacs. To do so, follow these steps:
  Go to the beginning of the document
  Type: M-x replace-string RET C-q C-m RET RET
  where "RET" means <press the return key> and C-q and C-m mean <hold the CTRL key and press the m (or q) key>.

see also

• https://its.ucsc.edu/unix-timeshare/tutorials/clean-ctrl-m.html

Mac OS: remove all routes

sudo route -n flush
sudo route add default 192.168.20.1

To list routing table on Mac OS:

netstat -rn

See also

• How to See the Routing Table on Mac OSX
• Network stops working, with "Can't Assign Requested Address"

Install openconnect on Mac OS

Install openconnect and openconnect-sso

brew install openconnect
pip3 install pyqt5 pyqtwebengine openconnect-sso

Connect to VPN

openconnect-sso -s your.vpn/address

Uninstall openconnect and openconnect-sso

brew uninstall openconnect
pip3 uninstall PyQt5-Qt5 PyQt5-sip PyQtWebEngine-Qt5 PySocks attrs colorama importlib-metadata keyring lxml openconnect-sso prompt-toolkit pyqt5 pyqtwebengine pyxdg structlog toml wcwidth zipp

Mount SMB share on Mac OS

mkdir ~/share1
mount_smbfs 'smb://DOMAIN;USERNAME:PASSWORD@smb-server.your.org/share1' ~/share1

or

mkdir ~/share1
mount_smbfs 'smb://DOMAIN;USERNAME@smb-server.your.org/share1' ~/share1

Properly kill ssh-agent on Linux

• eval "$(ssh-agent -k)"

• SSH_AGENT_PID="$(pidof ssh-agent)" ssh-agent -k

• kill -9 $(pidof ssh-agent)

see also

• How to kill ssh-agent properly on Linux

Mac OS: convert .mov file to .mp4

brew install ffmpeg
ffmpeg -i ./video1.mov ./video1.mp4

Git end-of-line conversion settings

Enable end-of-line conversion

git config --global core.autocrlf input

• Make sure your IDE (Text) Editors are unix/dos EOL aware.
• Global git config:

  git config --global core.autocrlf input
  git config --global core.safecrlf true

• Create .gitattributes file at the root the git repository, and add configuration like below:

  *       text=auto
  *.java      text
  *.xml       text
  *.sh        text eol=lf
  *.cmd       text eol=crlf
  *.bat       text eol=crlf
  *.jpg       -text
  *.png       -text
  *.gif       -text

• Normalise the EOL for all the files in the repository:

  git add --renormalize .

  git status        # Show files that will be normalized

  git commit -m "Introduce end-of-line normalization"

Disable end-of-line conversion

git config --global core.autocrlf false

See also

• Configure End-of-line conversion in .gitattributes file
• Mind the End of Your Line
• Git core.safecrlf different behavior on files with same line endings

Convert svg file to png on Mac OS

qlmanage -t -s 600 -o . logo.svg 

It will produce logo.svg.png that is 600 pixels wide.

Mac OS: get file size using stat command

stat -f%z ./my_file

Execute a command on the results of find command

• execute the command multiple times on each individual result file:

  find . -type f -exec echo {} \;

  The result should look like:

  ./1
  ./3
  ./2

• execute the command only once on all result files:

  find . -type f -exec echo {} +

  The result should look like:

  ./1 ./3 ./2

Example: Remove .DS_Store and ._* files:

find . -type f -name '.DS_Store' -exec rm -f {} \;
find . -type f -name '._*' -exec rm -f {} \;

see also

• How to run find -exec

Remove ._ and .DS_Store files on Linux

find . -iname '._*' -exec rm -f {} \;
find . -iname '.DS_Store' -exec rm -f {} \;

find . -iname "._*" -delete
find . -iname ".DS_Store" -delete

Secure SSH key authentication with passphrase and ssh-agent

Secure your SSH key with passphrase

• Generate your SSH key pair for the remote SSH servers:

  ssh-keygen -t rsa -b 4096 -f ~/.ssh/gitlab_rsa

  
  Note: To secure your key pair, you must set a passphrase.
  It will create two files: gitlab_rsa and gitlab_rsa.pub in ~/.ssh directory.
  
• Configure to use the key pair when connecting to the remote SSH server by adding following lines to ~/.ssh/config

  Host gitlab.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/gitlab_rsa

  
  Now if you run git pull for the gitlab repositories. It will ask for the passphrase for gitlab_rsa key.

Use ssh-agent

If you want to do ssh or git pull without entering passphrase. You can use ssh-agent:

• Start ssh-agent:

  eval $(ssh-agent -s)

• Add the private key:

  ssh-add ~/.ssh/gitlab_rsa

  You need enter the passphrase to add the key.
• To list the keys managed by ssh-agent:

  ssh-add -l

By doing above, the agent will keep the private key in memory, and you do not need to enter the passphrase again.

See also

• SSH key authentication for gitlab
• https://stackoverflow.com/questions/17846529/could-not-open-a-connection-to-your-authentication-agent