If you want to run a script, ALWAYS log script output into a file or you will be bitten in the ass and would not have any log data for future reference.

In a regular whay when you try to run a python script you would use:

python my_script.py

Anything that will be printed inside of the script will be printed out into the stdout, so you use the below code to put the script output (stdout) into a file:

python my_script.py >> my_script.log

The above command will put the output into a persisted file that can be referenced in the future.

NOTE: The above scenario is for cases when you don't use a log handler in your script, or when
you are in a hurry and just want to put output in a file. Logging solution is definitely a good s
olution.

Finally if you want to run the script in background use:

python my_script.py >> my_script.log 2>&1

2>&1: 1 is for stdin and 2 is for stderr( if exist code of non-success happens). This command s
ays that send stderr messages into stdout.

#linux #python #script #log

Get current directory from within the bash script:

SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
 echo "$SCRIPT_DIR"

dirname gets the current directory and cd will change the current directory, finally pwd will return the current working directory, which in our case will be stored in SCRIPT_DIR.

#linux #bash #script #shell #pwd #current_directory

Delete elasticsearch indexes older than 1 month:

#!/bin/bash

 last_month=`date +%Y%m%d --date '1 month ago'`
 old_es_index="faxplus_*-$last_month"
 echo "Deleting ES indexes $old_es_index..."
 curl -X DELETE 'http://localhost:9200/myindex_*-20180520'
 echo ''

NOTE: asterisk in curl command will be anything in between of myindex_ and -20180520. For example myindex_module1-20180520.

#linux #sysadmin #bash #script #es #elasticsearch #DELETE #purge

https://unix.stackexchange.com/questions/31414/how-can-i-pass-a-command-line-argument-into-a-shell-script

#shell #argument #pass_argument #command_line #terminal #linux #bash #script

space=$(df -k / | tail -1 | awk '{print $4}')
 echo "free disk space is $space"

 if [ $space -lt 510000 ]
 then
   echo $(date) + " - Purge elasticsearch indexes..."
   curl -X DELETE "http://localhost:9200/your_index_name_*"
   echo ''
 else
   echo $(date) + " - disk space seems OK"
 fi

Put this in a crontab and you are good to go.

#linux #sysadmin #bash #script #df #elasticsearch #es

tail command in Linux is used to see content of a file from the end. It is usually used for checking log files in server. The interesting thing about tail is that you can use this command to get the last line. So in a bash script if you want to get last row of the below output:

root@server:~# ls -l
 total 24
 -rw-r--r-- 1 root root  291 May 26 05:19 es_queries
 -rw-r--r-- 1 root root 1198 Jun 19 10:34 users.json
 -rwxr-xr-x 1 root root  272 Jun 19 11:22 monitor_disk_space.sh
 -rwxr-xr-x 1 root root  433 Jun 19 10:00 another_script.sh

You would do:

root@server:~# ls -l | tail -1
 -rwxr-xr-x 1 root root  433 Jun 19 10:00 another_script.sh

That's why we have used this command in the previous post on df -k /.

#bash #tail #script #ls

Get the oldest elasticsearch index:

curl 'http://127.0.0.1:9200/_cat/indices' 2>&1 | awk '{print $3}' | grep "logstash_.*" | sort -t- -k2

DO NOT PANIC! Just enjoy it :)

First of all we use curl to get list of indexes from elasticsearch. By using awk with fetch just the 3rd column of the output, 3rd column refers to your index names (be careful to give your index name as there are internal indexes too and we do not want to purge them). grep command will then filter indexes and outputs those that start by logstash_, if yours are different change it. Finally the sort command sorts the result, but it first gets a delimiter by -t. sort -t- will split the column to TWO columns based on dash (-):

If my index name is logstash_data-20180619, it will exports 2 columns one is logstash_data and the other is 20180619. Now we use -k2 in order to sort based on the second column which is the date of the index.

This is how we can get the oldest elastic search index. I use this for maintenance of ES. In case disk space is almost full, I will delete the oldest elasticsearch index. You can even send a SLACK notification using cURL too.

The possibilities are endless.

Happy bashing :)

#linux #bash #curl #grep #sort #es #elasticsearch #split #awk #script

In Linux bash scripting you can check commands exit codes and do appropriate jobs accordingly. For that we will use || and &&.

Let's start by a simple echo command:

echo "Hello everybody"

If for any reason we want to check the exit code of echo command to see if it is successful or not. We can use the code block:

echo "Hello everybody" && echo "Phew! We're good." || echo "echo command FAILED!"

You can use code block to run multiple commands:

echo "Hello everybody" && {
   echo "Phew! We're good."
   touch ME
 } || {
   echo "echo command FAILED!"
   touch YOURSELF
 }

NOTE: exit code 0 means command execution was successful, and exit code 1 means something nasty happened to the previous command.


The is another way that you can check exit code and it is $?:

cp ME YOURSELF
 if [ $? = 0 ] ; then
   echo "copy seems OK!"
 else
   echo "Yuck! File could not get copied! :("
 fi

When cp command is run $? will keep the exit code of recent command which has been executed.

#linux #bash #script #scripting #exit_code

In bash script you can get count of given arguments to your script using $#. If you want to get a positional argument use $ and then number like $1. If you want to get the exit code of the previous command use $?.

#linux #bash #script

How to prepend a string to all file names in a directory in a bash script?

for f in *.py; do mv "$f" "old-$f"; done

The above one-line will loop over all files in current directory with .py extension and prepend old- into the files.

So for example a file with name main.py will be renamed to old-main.py

#python #bash #script #prepend #move #rename #for

I sometimes forgot to pull data from git before start working on a project. To minimize the headache of merge conflict or having to reset head and stash your data you can set a cronjob to run every 10 minute or so. And inside of your bash script CD into eaach of your folders and issue git pull command.

If you have similar issues and you're solving it in a different manner, I'd be happy to hear about it. :)

#linux #mac #bash #script #git #cron #crontab #cronjob

Install & configure gitlab-runner:

https://gist.github.com/muellermarkus/76a02f6fa46206d4ec61c5c0af9f357c

#gitlab #git #runner #runner #gitlab_runner #script #sh #gist #github