In python when you open a file using open command the your can read the content of the file. read will read the whole content of the file at once, while readline reads the content of the file line by line.

NOTE: if file is huge, read() is definitely a bad idea, as it loads (without size parameter), whole file into memory.

NOTE: it is good practice to use the with keyword when dealing with file objects. The advantage is that the file is properly closed after its suite finishes, even if an exception is raised at some point. (We have reviewed with in depth a couple days ago)

NOTE: read function gets a size parameter that specifies the chunks read from a file. If the end of the file has been reached, f.read() will return an empty string.

For reading lines from a file, you can loop over the file object. This is memory efficient, fast, and leads to simple code:

>>>
 >>> for line in f:
 ...     print(line, end='')
 ...
 This is the first line of the file.
 Second line of the file

#python #file #read #readline #efficiency

1. List all Open Files with lsof Command

> lsof
 COMMAND    PID      USER   FD      TYPE     DEVICE  SIZE/OFF       NODE NAME
 init         1      root  cwd      DIR      253,0      4096          2 /
 init         1      root  rtd      DIR      253,0      4096          2 /
 init         1      root  txt      REG      253,0    145180     147164 /sbin/init
 init         1      root  mem      REG      253,0   1889704     190149 /lib/libc-2.12.so

FD column stands for File Descriptor, This column values are as below:
- cwd current working directory
- rtd root directory
- txt program text (code and data)
- mem memory-mapped file


To get the count of open files you can use wc -l with lsof like as follow:

lsof | wc -l

2. List User Specific Opened Files

lsof -u alireza
 COMMAND  PID    USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
 sshd    1838 alireza  cwd    DIR      253,0     4096      2 /
 sshd    1838 alireza  rtd    DIR      253,0     4096      2 /

#linux #sysadmin #lsof #wc #file_descriptor

In Linux we have a command called test, you can check whether a file/directory exists or not and run commands based on the result. For example let's say we want to check if a folder exists and if it does not exist, create the folder.

For checking directory existence we use test -d and for file existence we use test -f, so for our example in order to check if the directory exists we use test -d and in case the folder does not exists we will create it:

directory_to_check="/data/mysql"
 test -d $directory_to_check || {
     echo "$directory_to_check does not exist, creating the folder..." && mkdir -p $directory_to_check || {
         echo "$directory_to_check directory could not be created!"
         exit 1
     }
 }

NOTE: you can read more about exit codes with hashtag #exit_code

#bash #linux #directory_existence #file_existence

If your pyCharm does not recognize specific python files as a Python file and does not provide syntax highlighting for you, you need to navigate to File > Settings > Editor > File Types > Text and Under Registered Patterns, you can see the new myfilename.py in the list. Remove it from the list with the - button and click OK at the end.

#python #pycharm #file_types #syntax_highlighting

How to upload file into Amazon object storage using boto3?

pip install boto3

Now you just need the region, endpoint and access key, secret key which you would be given after purchase:

client = session.client('s3',
                         region_name=YOUR_REGION,
                         endpoint_url=YOUR_HOST,
                         aws_access_key_id=YOUR_ACCESS_KEY,
                         aws_secret_access_key=YOUR_SECRET_KEY)

 client.upload_file(file_path,  # Path to local file
                    obj_config['spacename'],  # Name of Space
                    'YOUR_FILE_NAME.txt',  # Name for remote file
                     ExtraArgs={"Metadata": {'user-id': USER_ID} })  # metadata

NOTE: in the name of the file you can pass / like my/file/here.txt. Now it will create directory (virtually) in the remote object storage.

#python #object_storage #boto3 #file_upload

https://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto

#python #encryption #file_encryption #pycrypto #AES #IV #CBC

https://medium.com/@nielssj/docker-volumes-and-file-system-permissions-772c1aee23ca

#medium #docker #volumes #file_system