What are cat and zcat commands and what is their differences?

cat is used to print the content of a file (in stdout or a file, etc). It's main usage is when you want to search something in the whole file like a log file:

sudo cat /var/log/nginx/access.log | grep --color "SOMETHING TO SEARCH"

zcat command on the other is used to get the content of a .gz compressed text file like apache.log.1.gz:

sudo zcat /var/log/apache/apache.log.1.gz | grep --color "SOMETHING TO SEARCH"

NOTE: grep command is used to search in a file. the symbol | (pipeline) is used to send (pipe) first command result into the second command.

#linux #grep #cat #zcat #command

When you see logs in docker you cannot use grep on the output. In case you want to put grep on it you need to send data to standard output (2>&1).

Long story, short:

docker service logs --since "1m" -f app_redis 2>&1 | grep "Your search text"

#docker #log #logs #since #grep

If you grep a keyword in a text file and wants to print line number of the found keyword use -n with your grep command:

grep -n "hello world" *

#linux #grep #line_number

Get the oldest elasticsearch index:

curl 'http://127.0.0.1:9200/_cat/indices' 2>&1 | awk '{print $3}' | grep "logstash_.*" | sort -t- -k2

DO NOT PANIC! Just enjoy it :)

First of all we use curl to get list of indexes from elasticsearch. By using awk with fetch just the 3rd column of the output, 3rd column refers to your index names (be careful to give your index name as there are internal indexes too and we do not want to purge them). grep command will then filter indexes and outputs those that start by logstash_, if yours are different change it. Finally the sort command sorts the result, but it first gets a delimiter by -t. sort -t- will split the column to TWO columns based on dash (-):

If my index name is logstash_data-20180619, it will exports 2 columns one is logstash_data and the other is 20180619. Now we use -k2 in order to sort based on the second column which is the date of the index.

This is how we can get the oldest elastic search index. I use this for maintenance of ES. In case disk space is almost full, I will delete the oldest elasticsearch index. You can even send a SLACK notification using cURL too.

The possibilities are endless.

Happy bashing :)

#linux #bash #curl #grep #sort #es #elasticsearch #split #awk #script

I have a script that checks a source folder for new files in case there are files in the source folder, it will move those files to destination.

The problem I encountered recently was that files are huge and it may be in the middle of the copying into source by another process so my script tries to move an incomplete file to a destination. Let's say the file is 4GB in size and just only 1GB of the file has been copied. I have to wait until file is 4GB and other handler using that file, then I should safely move the file.

You can use lsof command in order to check which processes are using the source file:

if [[ `lsof -- /var/my-folder/my-big-file.tar.gz` ]]
then
     echo "File is being used by a process."
     exit 1
 fi

NOTE: you can give file directly to lsof using -- or you can use grep command as follow:

lsof | grep /var/my-folder/my-big-file.tar.gz

NOTE2: if you are in a loop use break instead of exit.

NOTE3: if you get command not found, install it using apt-get install lsof

#linux #sysadmin #lsof #grep

https://www.digitalocean.com/community/tutorials/using-grep-regular-expressions-to-search-for-text-patterns-in-linux

#linux #grep #regex