tail command in Linux is used to see content of a file from the end. It is usually used for checking log files in server. The interesting thing about tail is that you can use this command to get the last line. So in a bash script if you want to get last row of the below output:

root@server:~# ls -l
 total 24
 -rw-r--r-- 1 root root  291 May 26 05:19 es_queries
 -rw-r--r-- 1 root root 1198 Jun 19 10:34 users.json
 -rwxr-xr-x 1 root root  272 Jun 19 11:22 monitor_disk_space.sh
 -rwxr-xr-x 1 root root  433 Jun 19 10:00 another_script.sh

You would do:

root@server:~# ls -l | tail -1
 -rwxr-xr-x 1 root root  433 Jun 19 10:00 another_script.sh

That's why we have used this command in the previous post on df -k /.

#bash #tail #script #ls

Have you ever wanted to syntax highlight text area with specific content in it? Let's say it contains json data. We also need to have
code folding (+/- in front of objects to collapse them). The tool that can be used for this purpose is codeMirror:
- https://codemirror.net/index.html

One of the things that I want to note here is that textarea wont get updated when you enter data in codeMirror field. For that you need to call save() method of codeMirror like below:

var myEditor = CodeMirror.fromTextArea(page_content, {
     lineNumbers: true,
     mode: "markdown",
     lineWrapping: true,
     lineNumbers:false,
     indentWithTabs: true
 });

 function updateTextArea() {
     myEditor.save();
 }
 myEditor.on('change', updateTextArea);

Download all the demos from github:
- https://github.com/codemirror/codemirror


Instead of myEditor change you can update textarea on form submit.

#syntax_highlighting #syntax #codeMirror #code_folding

#programmer_jokes #fun

pyflame, a fantastic python profiler that uses linux ptrace system call to collect profiling information. It gives you a graph to see where you have messed things up!

One of the great great great things about this library is that you can attach it to a currently running process to profile it. A command like below will do the job:

# Attach to PID 12345 and profile it for 1 second
 pyflame -p 12345

Code and installation can be found in github:
- https://github.com/uber/pyflame

Read more about it:
- https://pyflame.readthedocs.io/en/latest/

#python #pyflame #profiler

In debian linux you can check a directory size by du -h -d 1, BUT in CentOS it is a little bit different:

du -sh /path/to/dir/*

#linux #du #centos #centos #debian

#programmer_jokes #fun

If for any reason you had to increase uwsgi_pass timeout in nginX you can use uwsgi_read_timeout:

upstream uwsgicluster {
     server 127.0.0.1:5000;
 }

 .
 .
 .


 include            uwsgi_params;
 uwsgi_pass         uwsgicluster;
 uwsgi_read_timeout 3000;

You can also increase timeout in uwsgi. If you are using ini file you need to use harakiri parameter like below:

harakiri = 30

Its value is in seconds.

#uwsgi #nginx #uwsgi_pass #harakiri #timeout #uwsgi_read_timeout

Python offers count method that be used in order to see how many of a specific character exists in a string:

> number = '+982111111111****1'
 > number.count('*')
 4

That's how it works. Great tiny tool.

#python #count

http://zdk.blinkenshell.org/redis-backup-and-restore/

#redis #backup #restore

Run a linux command multiple times:

for i in `seq 10`; do command; done

Or equivalently, using the Bash builtin for generating sequences:

for i in {1..10}; do command; done

#linux #bash #seq #repeat

Get the oldest elasticsearch index:

curl 'http://127.0.0.1:9200/_cat/indices' 2>&1 | awk '{print $3}' | grep "logstash_.*" | sort -t- -k2

DO NOT PANIC! Just enjoy it :)

First of all we use curl to get list of indexes from elasticsearch. By using awk with fetch just the 3rd column of the output, 3rd column refers to your index names (be careful to give your index name as there are internal indexes too and we do not want to purge them). grep command will then filter indexes and outputs those that start by logstash_, if yours are different change it. Finally the sort command sorts the result, but it first gets a delimiter by -t. sort -t- will split the column to TWO columns based on dash (-):

If my index name is logstash_data-20180619, it will exports 2 columns one is logstash_data and the other is 20180619. Now we use -k2 in order to sort based on the second column which is the date of the index.

This is how we can get the oldest elastic search index. I use this for maintenance of ES. In case disk space is almost full, I will delete the oldest elasticsearch index. You can even send a SLACK notification using cURL too.

The possibilities are endless.

Happy bashing :)

#linux #bash #curl #grep #sort #es #elasticsearch #split #awk #script

Simple bash script to take nightly MongoDB backups:

#!/bin/sh
 DIR=`date +%m%d%y`
 DEST=/db_backups/$DIR
 mkdir $DEST
 mongodump -h <your_database_host> -d <your_database_name> -u <username> -p <password> -o $DEST

NOTE: db_backups folder shoud already be created by mkdir /db_backups.


Put it in a crontab for nightly backups. First open crotab:

sudo crontab -e

Create a new line (entry) in crontab and paste the below cron task:

45 1 * * * ../../scripts/db_backup.sh

NOTE: here our script is called db_backup.sh, should you use your own script name here. and make it executable by chmod +x /your/ full_path/scripts/db_backup.sh


#mongodb #backup #cron #cronjob #coderwall #mongodump #bash

In order to compress a file with level of 9 (maximum level of compression), you need to
set an ENV variable. In order to not clobber the file system environments you can use pipe in your command:

tar cvf - /path/to/directory | gzip -9 - > file.tar.gz

#tar #gzip #compression_level

#redis backup script

https://gist.github.com/alirezastack/86530ae1a41e516057df71607ea08299

In Linux bash scripting you can check commands exit codes and do appropriate jobs accordingly. For that we will use || and &&.

Let's start by a simple echo command:

echo "Hello everybody"

If for any reason we want to check the exit code of echo command to see if it is successful or not. We can use the code block:

echo "Hello everybody" && echo "Phew! We're good." || echo "echo command FAILED!"

You can use code block to run multiple commands:

echo "Hello everybody" && {
   echo "Phew! We're good."
   touch ME
 } || {
   echo "echo command FAILED!"
   touch YOURSELF
 }

NOTE: exit code 0 means command execution was successful, and exit code 1 means something nasty happened to the previous command.


The is another way that you can check exit code and it is $?:

cp ME YOURSELF
 if [ $? = 0 ] ; then
   echo "copy seems OK!"
 else
   echo "Yuck! File could not get copied! :("
 fi

When cp command is run $? will keep the exit code of recent command which has been executed.

#linux #bash #script #scripting #exit_code

Delete files older than X days. You can use find command in order to find files with specific patterns in a specific directory and then remove those files:

BACKUP_DIR=/var/backup

 # Number of days to keep backups
 KEEP_BACKUPS_FOR=30 #days

 find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +$KEEP_BACKUPS_FOR -exec rm {} \;

-mtime n[smhdw]:
If no units are specified, this primary evaluates to true if the difference between the file last modification time and the time find was started, rounded up to the next full 24-hour period, is n 24-hour periods.

-exec utility [argument ...] ;
True if the program named utility returns a zero value as its exit status. Optional arguments may be passed to the utility. The expression must be terminated by a semicolon (``;''). If you invoke find from a shell you may need to quote the semicolon if the shell would otherwise treat it as a control operator. If the string ``{}'' appears anywhere in the utility name or the arguments it is replaced by the pathname of the current file. Utility will be executed from the directory from which find was executed. Utility and arguments are not subject to the further expansion of shell patterns and constructs. SO BECAREFUL SEMICOLON IS NEEDED!

#find #rm #exec #mtime #remove_old_files #remove

In Linux we have a command called test, you can check whether a file/directory exists or not and run commands based on the result. For example let's say we want to check if a folder exists and if it does not exist, create the folder.

For checking directory existence we use test -d and for file existence we use test -f, so for our example in order to check if the directory exists we use test -d and in case the folder does not exists we will create it:

directory_to_check="/data/mysql"
 test -d $directory_to_check || {
     echo "$directory_to_check does not exist, creating the folder..." && mkdir -p $directory_to_check || {
         echo "$directory_to_check directory could not be created!"
         exit 1
     }
 }

NOTE: you can read more about exit codes with hashtag #exit_code

#bash #linux #directory_existence #file_existence

With mysqldump you can export databases. with --port parameter you can specify which port it should connects. If you provide localhost for --host parameter, mySQL will use sockets and port will be ignored.

So be careful with it!

#mysql #mysqldump #port #port_ignorance #3306 #backup #database_backup #sockets #ip_address #localhost

https://www.shellcheck.net/

#linux #shell #shell_check #bash #online_shell_check

An easy way to encrypt and decrypt large files using OpenSSL and Linux:

Generate PEM public private key using openssl:

openssl req -x509 -nodes -newkey rsa:2048 -keyout private-key.pem -out public-key.pem

Encrypt file using public key PEM file:

openssl smime -encrypt -binary -aes-256-cbc -in large_file.img -out large_file.img.dat -outform DER public-key.pem

We can generate hash using md5sum for both files so we can compare them once we decrypt our file:

md5sum large_file.img*
 #cd573cfaace07e7949bc0c46028904ff  large_file.img
 #c4d8f1e868d1176d8aa5363b0bdf8e7c  large_file.img.dat

Decrypt large file using OpenSSL:

openssl smime -decrypt -in large_file.img.dat -binary -inform DEM -inkey private-key.pem -out decrypted_large_file.img

Check md5sum output:

md5sum *large_file.img*
 #cd573cfaace07e7949bc0c46028904ff  decrypted_large_file.img
 #cd573cfaace07e7949bc0c46028904ff  large_file.img
 #c4d8f1e868d1176d8aa5363b0bdf8e7c  large_file.img.dat

#linux #openssl #pem #encryption #decryption #x509 #public_key #private_key