MYSQL insert

If you have bulk scripts for your email/sms like me, and you are sending to thousands of users, you will difintely will get stuck in the middle of the bulk notification or it will be very slow and unacceptable. First of all you must initiate only ONE mysql connection. If you are inserting your data one by one, you're dead again! try to use executemany that will insert data into mySQL in bulk not one by one:

client.executemany(
       """INSERT INTO email (name, spam, email, uid, email_content)
       VALUES (%s, %s, %s, %s, %s)""",
       [
       ("Ali", 0, 'mail1@example.com', 1, 'EMAIL_CONTENT'),
       ("Reza", 1, 'mail2@example.com', 2, 'EMAIL_CONTENT'),
       ("Mohsen", 1, 'mail3@example.com', 3, 'EMAIL_CONTENT')
       ] )

Other note for bulk insertion is to avoid disk IO in case possible, and use redis, memcached or so on for inserting some data like user's phone or emails. It will tremendously improve performance of your bulk script.

#python #mysql #executemany #redis #bulk #email #sms

Transactions in Redis

MULTI, EXEC, DISCARD and WATCH are the foundation of transactions in Redis. They allow the execution of a group of commands in a single step, with two important guarantees:

- All the commands in a transaction are serialized and executed sequentially. It can never happen that a request issued by another client is served in the middle of the execution of a Redis transaction. This guarantees that the commands are executed as a single isolated operation.

- Either all of the commands or none are processed, so a Redis transaction is also atomic. The EXEC command triggers the execution of all the commands in the transaction, so if a client loses the connection to the server in the context of a transaction before calling the MULTI command none of the operations are performed, instead if the EXEC command is called, all the operations are performed. When using the append-only file Redis makes sure to use a single write(2) syscall to write the transaction on disk. However if the Redis server crashes or is killed by the system administrator in some hard way it is possible that only a partial number of operations are registered. Redis will detect this condition at restart, and will exit with an error. Using the redis-check- aof tool it is possible to fix the append only file that will remove the partial transaction so that the server can start again.


Sample usage of the transaction:

> MULTI
 OK
 > INCR foo
 QUEUED
 > INCR bar
 QUEUED
 > EXEC
 1) (integer) 1
 2) (integer) 1

As it is possible to see from the session above, EXEC returns an array of replies, where every element is the reply of a single command in the transaction, in the same order the commands were issued.

In the next post we will talk about WATCH and DISCARD commands too.

#redis #transaction #multi #exec #discard #watch

Transactions in Redis part2

DISCARD can be used in order to abort a transaction. In this case, no commands are executed and the state of the connection is restored to normal.


We can discard a transaction like below:

> SET foo 1
 OK
 > MULTI
 OK
 > INCR foo
 QUEUED
 > DISCARD
 OK
 > GET foo
 "1"

As you can see foo variable has not been incremented and value is set to 1 not 2.


Optimistic locking using check-and-set:

WATCH is used to provide a check-and-set (CAS) behavior to Redis transactions.

WATCHed keys are monitored in order to detect changes against them. If at least one watched key is modified before the EXEC command, the whole transaction aborts, and EXEC returns a Null reply to notify that the transaction failed.

WATCH mykey
 val = GET mykey
 val = val + 1
 MULTI
 SET mykey $val
 EXEC

Using the above code, if there are race conditions and another client modifies the result of val in the time between our call to WATCH and our call to EXEC, the transaction will fail.


#redis #transaction #multi #exec #discard #watch

Transactions in Redis part3

In order to implement transaction in Python you need to use pipline and there is no such a thing as exec, multi, etc.

r = redis.Redis()
 p = r.pipeline()
 p.set("transError", var)
 p.execute()

MULTI, SET, EXEC sent when p.execute() is called. To omit the MULTI/EXEC pair, use r.pipeline(transaction=False).

More info: http://redis-py.readthedocs.io/en/latest/#redis.Redis.pipeline

#python #redis #transaction #multi #exec

Those who are looking for scaling their cache server horizontally (redis/memcached) can use twemproxy. It is created by twitter:

https://github.com/twitter/twemproxy

#cache #redis #memcached #twemproxy

There are times that you noway but getting data from a third part library and that library has rate limit on their endpoints. For example I have recently used geopy python library to get latitude and longitude by giving city name to the function:

from geopy.geocoders import Nominatim

 city_name = 'Tehran'
 geolocator = Nominatim()
 location = geolocator.geocode(city_name)
 print location.latitude, location.longitude

This library by default sends its requests to https://nominatim.openstreetmap.org/search to get geo location data. It's rate limit is 1 request per second. To circumvent these problems and limitations use redis to cache results in your server and read cached result from your own system:

self.redis.hset(city_name, 'lat', lat)
 self.redis.hset(city_name, 'long', longitude)

Now read from cache in case it exists:

if self.redis.hexists(city_name, 'lat'):
     location = self.redis.hgetall(city_name)

Make sure to put sleep(1) when reading from Nominatim in order to by pass its limitation.


NOTE: instead of Nominatim other 3rd parties can be used.

#python #geopy #geo #latitude #longitude #Nominatim #redis #hset #geocoders

http://zdk.blinkenshell.org/redis-backup-and-restore/

#redis #backup #restore

#redis backup script

https://gist.github.com/alirezastack/86530ae1a41e516057df71607ea08299

https://www.hugeserver.com/kb/install-redis-4-debian-9-stretch/

#debian #redis #redis4

To get an expiration time of a redis key you can use TTL like below:

TTL "YOUR_KEY_NAME"

To read more about setting expiration time and or other options:
- https://redis.io/commands/expire

#redis #expiration_time #expire #ttl