Turn MySQL table into utf8mb4 to store emojis:

ALTER TABLE YOUR_TABLE convert to character set utf8mb4 collate utf8mb4_general_ci;

Moreover you also need to change column character set:

ALTER TABLE YOUR_TABLE CHANGE YOUR_COLUMN_NAME YOUR_COLUMN_NAME  VARCHAR(250) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Be careful that now you have to do more things like set character set after connection initiation in Python:

your_mysql_client = MySQLdb.connect(...)
 your_mysql_client.set_character_set('utf8mb4')

Now before executing your query you also need to set character set on cursor:

my_cursor.execute("SET NAMES utf8mb4;")
 my_cursor.execute(YOUR_QUERY)

#database #mysql #character_set #utf8mb4 #cursor #emoji

Months ago we have talked about how to get mongoDB data changes. THe problem with that article was that if for any reason your script
was stopped you will lose the data in the downtime period.

Now we have a new solution that you will read from the point in time that have read last time. MongoDB uses bson Timestamp in order for its internal usage like replication oplog logs. We can use the same Timestamp and store it somewhere to read from the exact point
that we have read last time.

In python you can import it like below:

from bson.timestamp import Timestamp

Now to read data from that point read that time stamp from where you have saved it and query the oplog from that point:

ts = YOUR_TIMESTAMP_HERE
 cursor = oplog.find({'ts': {'$gt': ts}},
                   cursor_type=pymongo.CursorType.TAILABLE_AWAIT,
                   oplog_replay=True)

After traversing cursors and catching mongoDB changes you can store the new timestamp that resides in ts field in the document you have fetched from MongoDB oplog.

Now use a while True and read data until cursor is alive. The point of this post is that you can store ts somewhere and read from the point you have stored ts.


If you remember from before we got last changes by the query below:

last = oplog.find().sort('$natural', pymongo.DESCENDING).limit(1).next()
 ts = last['ts']

We read the last ts and read from the last record, that's why we were missing data.

#mongodb #mongo #replication #oplog #timestamp #cursor