If for any reason you had to increase uwsgi_pass timeout in nginX you can use uwsgi_read_timeout:

upstream uwsgicluster {
     server 127.0.0.1:5000;
 }

 .
 .
 .


 include            uwsgi_params;
 uwsgi_pass         uwsgicluster;
 uwsgi_read_timeout 3000;

You can also increase timeout in uwsgi. If you are using ini file you need to use harakiri parameter like below:

harakiri = 30

Its value is in seconds.

#uwsgi #nginx #uwsgi_pass #harakiri #timeout #uwsgi_read_timeout

Python offers count method that be used in order to see how many of a specific character exists in a string:

> number = '+982111111111****1'
 > number.count('*')
 4

That's how it works. Great tiny tool.

#python #count

http://zdk.blinkenshell.org/redis-backup-and-restore/

#redis #backup #restore

Run a linux command multiple times:

for i in `seq 10`; do command; done

Or equivalently, using the Bash builtin for generating sequences:

for i in {1..10}; do command; done

#linux #bash #seq #repeat

Get the oldest elasticsearch index:

curl 'http://127.0.0.1:9200/_cat/indices' 2>&1 | awk '{print $3}' | grep "logstash_.*" | sort -t- -k2

DO NOT PANIC! Just enjoy it :)

First of all we use curl to get list of indexes from elasticsearch. By using awk with fetch just the 3rd column of the output, 3rd column refers to your index names (be careful to give your index name as there are internal indexes too and we do not want to purge them). grep command will then filter indexes and outputs those that start by logstash_, if yours are different change it. Finally the sort command sorts the result, but it first gets a delimiter by -t. sort -t- will split the column to TWO columns based on dash (-):

If my index name is logstash_data-20180619, it will exports 2 columns one is logstash_data and the other is 20180619. Now we use -k2 in order to sort based on the second column which is the date of the index.

This is how we can get the oldest elastic search index. I use this for maintenance of ES. In case disk space is almost full, I will delete the oldest elasticsearch index. You can even send a SLACK notification using cURL too.

The possibilities are endless.

Happy bashing :)

#linux #bash #curl #grep #sort #es #elasticsearch #split #awk #script

Simple bash script to take nightly MongoDB backups:

#!/bin/sh
 DIR=`date +%m%d%y`
 DEST=/db_backups/$DIR
 mkdir $DEST
 mongodump -h <your_database_host> -d <your_database_name> -u <username> -p <password> -o $DEST

NOTE: db_backups folder shoud already be created by mkdir /db_backups.


Put it in a crontab for nightly backups. First open crotab:

sudo crontab -e

Create a new line (entry) in crontab and paste the below cron task:

45 1 * * * ../../scripts/db_backup.sh

NOTE: here our script is called db_backup.sh, should you use your own script name here. and make it executable by chmod +x /your/ full_path/scripts/db_backup.sh


#mongodb #backup #cron #cronjob #coderwall #mongodump #bash

In order to compress a file with level of 9 (maximum level of compression), you need to
set an ENV variable. In order to not clobber the file system environments you can use pipe in your command:

tar cvf - /path/to/directory | gzip -9 - > file.tar.gz

#tar #gzip #compression_level

#redis backup script

https://gist.github.com/alirezastack/86530ae1a41e516057df71607ea08299

In Linux bash scripting you can check commands exit codes and do appropriate jobs accordingly. For that we will use || and &&.

Let's start by a simple echo command:

echo "Hello everybody"

If for any reason we want to check the exit code of echo command to see if it is successful or not. We can use the code block:

echo "Hello everybody" && echo "Phew! We're good." || echo "echo command FAILED!"

You can use code block to run multiple commands:

echo "Hello everybody" && {
   echo "Phew! We're good."
   touch ME
 } || {
   echo "echo command FAILED!"
   touch YOURSELF
 }

NOTE: exit code 0 means command execution was successful, and exit code 1 means something nasty happened to the previous command.


The is another way that you can check exit code and it is $?:

cp ME YOURSELF
 if [ $? = 0 ] ; then
   echo "copy seems OK!"
 else
   echo "Yuck! File could not get copied! :("
 fi

When cp command is run $? will keep the exit code of recent command which has been executed.

#linux #bash #script #scripting #exit_code

Delete files older than X days. You can use find command in order to find files with specific patterns in a specific directory and then remove those files:

BACKUP_DIR=/var/backup

 # Number of days to keep backups
 KEEP_BACKUPS_FOR=30 #days

 find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +$KEEP_BACKUPS_FOR -exec rm {} \;

-mtime n[smhdw]:
If no units are specified, this primary evaluates to true if the difference between the file last modification time and the time find was started, rounded up to the next full 24-hour period, is n 24-hour periods.

-exec utility [argument ...] ;
True if the program named utility returns a zero value as its exit status. Optional arguments may be passed to the utility. The expression must be terminated by a semicolon (``;''). If you invoke find from a shell you may need to quote the semicolon if the shell would otherwise treat it as a control operator. If the string ``{}'' appears anywhere in the utility name or the arguments it is replaced by the pathname of the current file. Utility will be executed from the directory from which find was executed. Utility and arguments are not subject to the further expansion of shell patterns and constructs. SO BECAREFUL SEMICOLON IS NEEDED!

#find #rm #exec #mtime #remove_old_files #remove

In Linux we have a command called test, you can check whether a file/directory exists or not and run commands based on the result. For example let's say we want to check if a folder exists and if it does not exist, create the folder.

For checking directory existence we use test -d and for file existence we use test -f, so for our example in order to check if the directory exists we use test -d and in case the folder does not exists we will create it:

directory_to_check="/data/mysql"
 test -d $directory_to_check || {
     echo "$directory_to_check does not exist, creating the folder..." && mkdir -p $directory_to_check || {
         echo "$directory_to_check directory could not be created!"
         exit 1
     }
 }

NOTE: you can read more about exit codes with hashtag #exit_code

#bash #linux #directory_existence #file_existence

With mysqldump you can export databases. with --port parameter you can specify which port it should connects. If you provide localhost for --host parameter, mySQL will use sockets and port will be ignored.

So be careful with it!

#mysql #mysqldump #port #port_ignorance #3306 #backup #database_backup #sockets #ip_address #localhost

https://www.shellcheck.net/

#linux #shell #shell_check #bash #online_shell_check

An easy way to encrypt and decrypt large files using OpenSSL and Linux:

Generate PEM public private key using openssl:

openssl req -x509 -nodes -newkey rsa:2048 -keyout private-key.pem -out public-key.pem

Encrypt file using public key PEM file:

openssl smime -encrypt -binary -aes-256-cbc -in large_file.img -out large_file.img.dat -outform DER public-key.pem

We can generate hash using md5sum for both files so we can compare them once we decrypt our file:

md5sum large_file.img*
 #cd573cfaace07e7949bc0c46028904ff  large_file.img
 #c4d8f1e868d1176d8aa5363b0bdf8e7c  large_file.img.dat

Decrypt large file using OpenSSL:

openssl smime -decrypt -in large_file.img.dat -binary -inform DEM -inkey private-key.pem -out decrypted_large_file.img

Check md5sum output:

md5sum *large_file.img*
 #cd573cfaace07e7949bc0c46028904ff  decrypted_large_file.img
 #cd573cfaace07e7949bc0c46028904ff  large_file.img
 #c4d8f1e868d1176d8aa5363b0bdf8e7c  large_file.img.dat

#linux #openssl #pem #encryption #decryption #x509 #public_key #private_key

In cryptography world there are different ways that you can encrypt data (files). We explain both here a bit:

1- In one of these methods you encrypt a file with a secret key. The other party has the same secret key in order to be able to decrypt data. This way of encryption and decryption is called Symmetric.

2- The other method is Asymmetric encryption. Asymmetric encryption involves the use of a public key and a private key. As can be guessed from the names, only the private key needs to be kept private. The public key is made available for anybody that wants to encrypt data to send to the owner of the public key. Using the usual names Alice and Bob as examples: If Bob wanted to send an encrypted file or email to Alice, Bob would encrypt the data using Alice's public key then send the data to Alice. Alice can then decrypt the data using her private key that only she has access to. Once Bob has encrypted the data, nobody but Alice can decrypt it. Note that a private key is usually protected by a passphrase, so even on her own Ubuntu box where the key resides, Alice will still need to enter a passphrase to "unlock" the private key. Bob will have his own public/private key pair so people can send encrypted data to him. So should Alice want to send encrypted data to Bob, she'd encrypt it using Bob's public key whilst Bob could then decrypt it using his own private key. The best asymmetric cryptographic algorithm supported by gpg is called RSA.


#cryptography #asymmetric #symmetric #rsa #gpg

https://www.tutonics.com/2012/11/gpg-encryption-guide-part-1.html?m=1

#gpg #assymetric #cryptography

If you have multiple statements and you want to time the whole statements in a bash script you can use time command, but you don't need to use inside of the script. If you have an script called my_long_tasks.sh you need to just:

$ time my_long_tasks.sh
 real    0m20.894s
 user    0m3.664s
 sys     0m0.452s

That's it. In case you want to just time part of your statements in your bash script then you need to:

start=`date +%s`
 stuff
 end=`date +%s`

 runtime=$((end-start))

#linux #bash #time

In bash script you can get count of given arguments to your script using $#. If you want to get a positional argument use $ and then number like $1. If you want to get the exit code of the previous command use $?.

#linux #bash #script

#gpg #mail #encryption