Tech C**P
14 subscribers
161 photos
9 videos
59 files
304 links
مدرس و برنامه نویس پایتون و لینوکس @alirezastack
Download Telegram
space=$(df -k / | tail -1 | awk '{print $4}')
echo "free disk space is $space"

if [ $space -lt 510000 ]
then
echo $(date) + " - Purge elasticsearch indexes..."
curl -X DELETE "http://localhost:9200/your_index_name_*"
echo ''
else
echo $(date) + " - disk space seems OK"
fi

Put this in a crontab and you are good to go.

#linux #sysadmin #bash #script #df #elasticsearch #es
tail command in Linux is used to see content of a file from the end. It is usually used for checking log files in server. The interesting thing about tail is that you can use this command to get the last line. So in a bash script if you want to get last row of the below output:

root@server:~# ls -l
total 24
-rw-r--r-- 1 root root 291 May 26 05:19 es_queries
-rw-r--r-- 1 root root 1198 Jun 19 10:34 users.json
-rwxr-xr-x 1 root root 272 Jun 19 11:22 monitor_disk_space.sh
-rwxr-xr-x 1 root root 433 Jun 19 10:00 another_script.sh

You would do:

root@server:~# ls -l | tail -1
-rwxr-xr-x 1 root root 433 Jun 19 10:00 another_script.sh
That's why we have used this command in the previous post on df -k /.

#bash #tail #script #ls
Run a linux command multiple times:

for i in `seq 10`; do command; done


Or equivalently, using the Bash builtin for generating sequences:

for i in {1..10}; do command; done

#linux #bash #seq #repeat
Get the oldest elasticsearch index:

curl 'http://127.0.0.1:9200/_cat/indices' 2>&1 | awk '{print $3}' | grep "logstash_.*" | sort -t- -k2

DO NOT PANIC! Just enjoy it :)

First of all we use curl to get list of indexes from elasticsearch. By using awk with fetch just the 3rd column of the output, 3rd column refers to your index names (be careful to give your index name as there are internal indexes too and we do not want to purge them). grep command will then filter indexes and outputs those that start by logstash_, if yours are different change it. Finally the sort command sorts the result, but it first gets a delimiter by -t. sort -t- will split the column to TWO columns based on dash (-):

If my index name is logstash_data-20180619, it will exports 2 columns one is logstash_data and the other is 20180619. Now we use -k2 in order to sort based on the second column which is the date of the index.

This is how we can get the oldest elastic search index. I use this for maintenance of ES. In case disk space is almost full, I will delete the oldest elasticsearch index. You can even send a SLACK notification using cURL too.

The possibilities are endless.

Happy bashing :)

#linux #bash #curl #grep #sort #es #elasticsearch #split #awk #script
Simple bash script to take nightly MongoDB backups:

#!/bin/sh
DIR=`date +%m%d%y`
DEST=/db_backups/$DIR
mkdir $DEST
mongodump -h <your_database_host> -d <your_database_name> -u <username> -p <password> -o $DEST

NOTE: db_backups folder shoud already be created by mkdir /db_backups.


Put it in a crontab for nightly backups. First open crotab:

sudo crontab -e


Create a new line (entry) in crontab and paste the below cron task:

45 1 * * * ../../scripts/db_backup.sh

NOTE: here our script is called db_backup.sh, should you use your own script name here. and make it executable by chmod +x /your/ full_path/scripts/db_backup.sh


#mongodb #backup #cron #cronjob #coderwall #mongodump #bash
In Linux bash scripting you can check commands exit codes and do appropriate jobs accordingly. For that we will use || and &&.

Let's start by a simple echo command:

echo "Hello everybody"


If for any reason we want to check the exit code of echo command to see if it is successful or not. We can use the code block:

echo "Hello everybody" && echo "Phew! We're good." || echo "echo command FAILED!"


You can use code block to run multiple commands:

echo "Hello everybody" && {
echo "Phew! We're good."
touch ME
} || {
echo "echo command FAILED!"
touch YOURSELF
}

NOTE: exit code 0 means command execution was successful, and exit code 1 means something nasty happened to the previous command.


The is another way that you can check exit code and it is $?:

cp ME YOURSELF
if [ $? = 0 ] ; then
echo "copy seems OK!"
else
echo "Yuck! File could not get copied! :("
fi

When cp command is run $? will keep the exit code of recent command which has been executed.

#linux #bash #script #scripting #exit_code
In Linux we have a command called test, you can check whether a file/directory exists or not and run commands based on the result. For example let's say we want to check if a folder exists and if it does not exist, create the folder.

For checking directory existence we use test -d and for file existence we use test -f, so for our example in order to check if the directory exists we use test -d and in case the folder does not exists we will create it:

directory_to_check="/data/mysql"
test -d $directory_to_check || {
echo "$directory_to_check does not exist, creating the folder..." && mkdir -p $directory_to_check || {
echo "$directory_to_check directory could not be created!"
exit 1
}
}

NOTE: you can read more about exit codes with hashtag #exit_code

#bash #linux #directory_existence #file_existence
If you have multiple statements and you want to time the whole statements in a bash script you can use time command, but you don't need to use inside of the script. If you have an script called my_long_tasks.sh you need to just:

$ time my_long_tasks.sh
real 0m20.894s
user 0m3.664s
sys 0m0.452s


That's it. In case you want to just time part of your statements in your bash script then you need to:

start=`date +%s`
stuff
end=`date +%s`

runtime=$((end-start))

#linux #bash #time
In bash script you can get count of given arguments to your script using $#. If you want to get a positional argument use $ and then number like $1. If you want to get the exit code of the previous command use $?.

#linux #bash #script
Did you know you can test bash scripts line by line? Well, bash -x is here to help:

$ bash -x your_script.sh
+ a=10
+ echo 10
10


The content of the bash script is:

#!/bin/bash

a=10
echo $a

#bash #sh #shell #scripting #debug #debugging
Array and loop in bash script

To define an array you can use a structure like below, be careful that we don't use comma in between:

dbs=( 'test1' 'test2' 'test3' )


Now to loop over the array elements use for:

for your_db in ${dbs[@]}
do
echo $your_db
done

This is it!

#bash #scripting #for #loop #array
How to prepend a string to all file names in a directory in a bash script?

for f in *.py; do mv "$f" "old-$f"; done

The above one-line will loop over all files in current directory with .py extension and prepend old- into the files.

So for example a file with name main.py will be renamed to old-main.py

#python #bash #script #prepend #move #rename #for
I sometimes forgot to pull data from git before start working on a project. To minimize the headache of merge conflict or having to reset head and stash your data you can set a cronjob to run every 10 minute or so. And inside of your bash script CD into eaach of your folders and issue git pull command.

If you have similar issues and you're solving it in a different manner, I'd be happy to hear about it. :)

#linux #mac #bash #script #git #cron #crontab #cronjob
How to zero-pad a number in bash?

printf is here to help :)

In order to zero-pad a number you need to use do like below:

your_number_var=1
output=$(printf "%02d" $your_number_var)
echo $output # 01

Here I have used %02d. the number 2 refers to numbers of padding and d refers to digit. So to zero-pad to 5 you can use %05d.
As simple as that.

#bash #printf #zeropad #zero-pad #zeropadding
If you forget to pull your projects from git in a regular interval and many users working on the same projects, then there is a solution for you!

Create a bash script file as follow and make it executable by chmod +x puller.sh:

puller.sh file content:

#!/bin/bash

echo 'Iterating over folders...'
for dir in *
do
test -d "$dir" && {
cd ${dir}
echo "git pull $dir"
git pull
cd ".."
} || {
echo "------> $dir is not a directory <-------"
}
done

NOTE: this file should reside in your folder's project root. In my case it is in /Your/Projects/Folder.

Now as a final step, put it in your crontab:

10 * * * * bash -c "cd /Your/Projects/Folder; bash puller.sh >> /var/log/git_pull_output.log"

#linux #git #pull #cronjob #crontab #cron #bash
We have talked before about how to get current month using the below line of code:

echo $(date +%m)

It prints out 01, 02, etc.

As per the GNU date manpage:

By default, date pads numeric fields with zeroes. The following
optional flags may follow '%':
- (hyphen) do not pad the field

So you can remove leading zero by hyphen as below:

echo $(date +%-m)

It prints out 1, 2, etc.

#linux #bash #date
In order to enable bash completion in Kubernetes you can usee the below command in linux bash:

source <(kubectl completion bash)


Now to test this enter the below command and you should see the completion:

kubectl cl<TAB>


It should be expanded to kubectl cluster-info.


#linux #bash #shell #kubernetes #kubectl
How to recursively rename filenames?

find . -name '*txt' -exec bash -c ' mv $0 ${0/brand-/category-}' {} \;


The above command renames txt files starting with brand- to category-.

#linux #bash #find #rename #batch_rename