infrasturctures

I’m been wondering what’s running behind the scenes. I see system admins talking about prices from time to time, so I take notes.

没吃过猪肉，还没见过猪跑吗

Case: linux mirror (Jan. 2025)

https://social.kernel.org/notice/AqmnBVqyAncjoVhdNw

mirrors.kernel.org transfers ~1PiB of data weekly. At Equinix Metal’s list price of $0.05 per GB, it would cost us roughly $200, 000 a month to operate these systems

Case: freedesktop’s gitlab servers (Jan. 2025)

A very insightful post from FDO’s system admin.

https://web.archive.org/web/20250203121213/https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/2011

Case: elxir.bootlin .. the source code indexer

At Bootlin, here are a few details about the server we’re using: As of July 2019, our Elixir service consumes 17 GB of data (supporting all projects), or for the Linux kernel alone (version 5.2 being the latest), 12 GB for indexing data, and 2 GB for the git repository.

We’re using an LXD instance with 8 GB of RAM on a cloud server with 8 CPU cores running at 3.1 GHz.
https://github.com/bootlin/elixir

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

site map

sorry for the mess…

MAIN:

● HOME and ABOUT
● POSTS : regular posts
● NOTES : technical notes I keep for myself
● SYS : a system programming wiki I maintain
● MUSIC : my music
● VIS : some photos or arts
● TL : some fedi timeline
● LOG : random shitposts
● IRL : boring irl stuffs
● RANTS : hot takes, rants …
● TAKE : thoughts
● RAND : also thoughts but more spontaneous
● CODE READING : … code reading
● VOLATILE : frequently updated, drafts
● BITS : journals

MISC / AD-HOC

● BOOKMARKS : stuffs to read

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

pv (pipe viewer) - manipulate pipe and stream

----------------------

● Project URL: https://www.ivarch.com/programs/pv.shtml
● Code snippets under GPL-3.0-or-later
● Text under CC-BY-SA-4.0

----------------------

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

bits worth learning

if(NULL == thing){}as a sanity check. When (by mistake) writing == as =, if(thing = NULL){} is error-prone yet valid C code.. while if(NULL = thing) is a compilation error.

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

bits 0x55 - CW 9 - 18, 2025 (WIP)

It has been a long time. But I finally got my sanity back (spelt: thesis handed in). Anyways…

§§§ AI create more X-Y problems

So I see people talk on social media :

“I want to remove all blank lines form a text document, AI tools {A,B} don’t work, C works, wow amazing, here is how.”

Honestly if you spare one minute searching the internet:

Search for how to remove blank lines with [editor X], and you will immediately get a straight-forward solution. (mind you, these are literally the first entry from google search.

“delete empty lines vim”

:g/^$/d

“delete empty lines vscode”

Ctrl-H -> use regular expressions -> ^$\n -> replace all

“delete empty lines emacs”

M-x flush-lines RET ^$ RET

“delete empty lines notepad++”

edit -> line operations -> remove empty lines

…

See? Instead of these you decide to paste your whole document to a chatbot and let it do it for you. How do you know it only removes the black lines?

So I say it again: AI creates more X-Y problems.

● You have a solid problem
● instead of solving the problem
● you go after “how to prompt the AI to do it for me”.

§§§ shitphone experience

I’m traveling. However my phone has a failing battery and I have no intention to pick a new one as my main phone—the old one can be revived with a new battery. I need some safety for my travel … so I bought a cheap cheap backup phone for 65 euros brand new. To be specific it is a Redmi A3.

I have literally zero expectation from that thing. As long as it connects to the internet I’m happy. But setting it up is absofuckinglutely painful.

It asks you a bunch of questions, as for a google login, as you to agree to tons of stuffs. Fine, I won’t give you sensitive information anyways.

Then it proceeds to “set up your phone”, which is, installing more than 20 of APPs, including shit games, shit social media, shit streaming service. None I’ve ever asked for, or given the option to opt-in/out. And this process takes like an HOUR. Not including me removing those spyware.

I guess installing those sponsored APPs is how they make this phone cheap.

§§§ Good Reads

An Attempt to Catch Up with JIT Compilers—The False Lead of Optimizing Inline Caches. (Poirier et. al.)
https://arxiv.org/pdf/2502.20547

Contemp Culture - aurynn shaw

Systemd as tragedy

Does X cause Y? An in-depth evidence review

Modern-Day Oracles or Bullshit Machines?
hmmm. perhaps a good one(s) for the non-techies to de-mystify the AI superstition. However it reads way too subjective and I definitely hate the format: it’s like an apple product page. Can’t you just give me a text wall?

§§§ num num num

I have accumulated quite a few to-reads at @bookmarks. I see interesting stuffs floating around that I don’t have the time to read. Now that I have some time I’ll spend some time chewing some of them.

● I have some bookmarks on “system tracing”, especially in the linux eco-system. I’m making a standalone for this

§§§ git practice

I’m thinking about adapting a new git workflow. Namely bare-repo + worktree

for a non-ad-hoc work (e.g. nya)

1. have a local clone into a bare repo (under ~/repos/nya)
2. for a master/feature/bugfix, check out the branch into its own worktree (eng. under ~/projs/nya/{dev,feat1,fix_xyz})
3. delete the worktree once the branch is commited

~/repos/                ~/proj/
    |_ repo1 ------+--> |_ repo1_bugfix
    |              +--> |_ repo1_feature2
    |                   |
    |_ repo2 (stale)    |
    |                   |
    |_ repo3 ---------> |_ repo3_dev

How to trasform an existing repo into bare one (tested, won’t break existing worktrees)

SO answer:
https://stackoverflow.com/a/2200662

In short:

1. commit your work if any, otherwise you will lose them
2. replace your repo content with content under .git

mv repo/.git repo.git
rm -rf repo
mv repo.git repo

3. config git into bare repo

cd repo
`git config --bool core.bare true`

4. optionally, git worktree repair/prune to repair/remove broken worktrees

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

python package management and virtual env

I’m always scared away by python dependecy management. I’m always pissed off when I’m trying to install a pip package only to conflict with the system managed ones. Now I fainally take the chance to take a look at venv.

§ How does python (pip) manage packages?

AW: they fuck you.

§ How does linux distro manage python packages?

AW: they fuck you.

§ the venv (python 3.3+)

# create
$ python -m venv &LTname>   # python 3.3
$ virtualenv &LTname>       # python 3

# activate
$ source envname/bin/activate

# deactive
$ deactivate  # (while in virtenv)

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

glossary

BACK TO INDEX
polymeraseAny of various enzymes that catalyze the formation of polymers of DNA or RNA using an existing strand of RNA or DNA respectively as a template.

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

bioshocks (pun intended!)

https://www.genome.gov/about-genomics/fact-sheets/DNA-Sequencing-Fact-Sheet

One new sequencing technology involves watching DNA polymerase molecules as they copy DNA - the same molecules that make new copies of DNA in our cells - with a very fast movie camera and microscope […]

Wait ….. what…. movie camera????

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

read

volatile page for stuffs I read. With some takeaways and comments.

DNA sequencing fact sheet
this is not helping at all.

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

the DWARF Debugging Information Format (V5)

This post is a hash of the fucking manual: https://dwarfstd.org/doc/DWARF5.pdf

----------------------

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

pleroma maintainece notes

§ step 0:

● always check the release pages for breaking changes.
● make sure you have proper backups
● use e.g. tmux so that your session is not interrupted by internet conditions

Important docs

● archwiki: upgrading postgresql
● pleroma doc: updating your instance
● pleroma doc: Backup/Restore/Move/Remove your instance
● pleroma doc: database maintenance tasks

§ backups
[+] click to expand a sample pleroma backup script
assuming pleroma is installed from source

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

!!! do not run scripts from internet without knowing what it does !!!

# CHANGE THIS TO YOUR OWN CONFIG
PLEROMA_DB=pleroma              # name of the database
BACKUP_DIR=/backups             # path to backup storage
PLEROMA_INSTALL=/opt/pleroma    # path to pleroma installation
PLEROMA_UPLOADS=/srv/pleroma    # optionally you can backup pleroma statics (FEs, emojis, uploads etc.)

POSTGRES_USER=postgres          # system user for postgresql,
                                # by default it would be "postgres"

# you may want a timestamp for automated backups
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)

# use a temporary dir to collect all the data to create the final archive.
DUMP_DIR=/tmp/pleroma_dump/$TIMESTAMP
mkdir -p $DUMP_DIR

# dump the database(s)
sudo pg_dump --username=$POSTGRES_USER -d $PLEROMA_DB --format=custom -f $DUMP_DIR/postgres.dump

# you may want to check the return code for the previous command.

# backup important pleroma configs (optionally you could backup the whole
# pleroma production installation (e.g. /opt/pleroma)
cp $PLEROMA_INSTALL/config/prod.secret.exs $DUMP_DIR/prod.secret.exs
cp $PLEROMA_INSTALL/config/setup_db.psql $DUMP_DIR/setup_db.psql

# create an archive
tar -czvf  $BACKUP_DIR/$TIMESTAMP.tar.gz -C $DUMP_DIR . -C $PLEROMA_UPLOADS . >> /dev/null

# check results and remove the temporary dir
if [ $? -eq 0 ]; then
    echo [$TIMESTAMP] archive created
    rm -rf $DUMP_DIR
fi

§ upgrade postgresql (+migration)

You need to manually migrate the postgresql database when updating the postgres by major version, e.g. from 14.x to 15.y; You don’t have to do this when updating from e.g. 14.x to 14.y

Supposing upgrading from 14.x to 15.y, you need to make sure the 14.x postgresql binary is still available when doing the migration.

archlinux provides the postgresql-old-upgrade package which provides the postgresql binary 1 version behind the main repo postgresql.

My notes on upgrading pg

Here are some quick notes, but make sure you know what you are doing.

● make sure you have a proper backup
● stop postgresql service: systemctl stop postgresql
● update the postgresql binary and make sure you keep the old binary (see above)
● run the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# prepare dirs for live data migration

mv /var/lib/postgres/data /var/lib/postgres/olddata
mkdir /var/lib/postgres/data /var/lib/postgres/tmp
chown postgres:postgres /var/lib/postgres/data /var/lib/postgres/tmp

# switch to postgres user
sudo -iu postgres

# go to tmp dir and perform migration
cd /var/lib/postgres/tmp

# initialize the "new" DB. You'll need the parameters that you used to
# intialize the postgresql db for the first time. (if you follow pleroma
# installation guide this should be empty.)
initdb -D /var/lib/postgres/data --PARAMETERS_USED_TO_INIT_DB

# perform migration:
pg_upgrade -b /opt/pgsql-PG_VERSION/bin -B /usr/bin -d /var/lib/postgres/olddata -D /var/lib/postgres/data

# restart postgresql

for archlinux

optionally, to avoid suprises, do not update postgresql packages.

warning: postgresql: ignoring package upgrade (16.3-4 => 17.4-1) │ warning: postgresql-libs: ignoring package upgrade (16.3-4 => 17.4-1) │ warning: postgresql-old-upgrade: ignoring package upgrade (15.7-3 => 16.6-2)

§ install (update) pleroma from source

https://docs-develop.pleroma.social/backend/administration/updating/

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

frequenctly used

Debian Package Tracker
https://tracker.debian.org/

git notes
https://sys.shrik3.com/tooling/git/git.html

gpg notes
https://sys.shrik3.com/tooling/gpg/commands.html

Debian wiki: Using OpenPGP subkeys in Debian development
https://wiki.debian.org/Subkeys

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

mikrokernel - intro

return to index

motivation of microkernel (vs. monolith)

● isolation of system (kernel) components, reduce complicity, security and resilience
● small TCB
● user-level system components: flexibility and extensibility

§ acronyms and things

● L4RE: L4 Runtime Environment
● UTCB : user-level control block
● recursive address spaces, flexpage

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

ECC memory and bitflip handling

GOAL: understand how ECC hardware works & how the event propogates to the operating system (e.g. via interrupts)

§ ECC hardware:

1. ECC memory controller
2. on-chip (internal) error-correction circuits on DRAM chips
3. EOS memory modules

§ Physical Format, Chips and Memory Rank

----------------------

§ Reference

MCE (Machine Check Exception)

Hardware Specs

● Intel SDM Volume 3, Chapter 15 Machine-Check Architecture

APEI

EINJ

kernel

Misc

● Rowhammer, https://en.wikipedia.org/wiki/Row_hammer

----------------------

§ Glossary

● MCE : Machine Check Exception
● CMCI : Corrected Machine-Check Interrupt
● SEC-DED : Single-Error-Correction, Double-Error-Detection. ECC hardware that is able to correct single bit flip and detect double big flips.

Physical formats:

● SIMM : Single In-line Memory Module
● DIMM : Dual In-line Memory Module
● SO-DIMM : Small Outline DIMM

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

0x01 - coherence introduction

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

0x02 - memory constency intro & sequential consistency

[back to index]

● Memory Reordering - Intro
● Sequential Consistency (SC)

----------------------

without further notes, all contents are copy-verbatim or derived from Chapter 3 of

A Primer on Memory Consistency and Cache Coherence (2nd Edition), Synthesis Lectures on Computer Architecture, Vijay Nagarajan, Daniel J.Sorin, Mark D.Hill and David A. Wold
https://pages.cs.wisc.edu/~markhill/papers/primer2020_2nd_edition.pdf

this page is under CC-BY-SA-4.0. The above notice must be preserved.

----------------------

§ Memory Reordering - Intro

Example of memory reordering: core 2 spins upon flag. Programmer may assume Core 2 reads r2 == NEW because Core 1 sets data before setting flag. However S1 and S2 could get reordered, thus Core 2 exits the spin and reads old data before NEW is stored.

Sequential execution (von Neumann) model generally requires that operations to the same address execute in the original program order. The reordering discussed only happen on access to different addresses.

How a core might reorder memory access.

● Store-store reordering
● Load-load reordering
● Load-store and store-load reordering

§ Sequential Consistency (SC)
hello

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

Memory Model and Formalization

since the formal language (+model checking, formal verification etc.) is new knowledge to me, I’d like to repeat them here for a quick reference.

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

linux kernel synchronization (WIP)

Main reference:

Understanding the Linux Kernel 3rd Edition, Daniel P.Bovet and Marco Cesati.

TODOs:

● preempt_count

General Rule
kernel preemption is disabled in critical region protected by spin locks

§ Quick facts

spin locks

● kernel preemption is disabled in spin lock protected critical regions.
● for uniprocessor system, the spin lock simply disables/enables kernel preemption

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

some atomic memory operations

CMPXCHG - Comapre and Exchange

    [lock] cmpxchg  reg, reg/mem
    ^               ^       ^
    |               |       |
    prefix          SRC     DEST

if al == dest:
    ZF      &LT- 1
    DEST    &LT- SRC
else:
    ZF      &LT- 0
    AX      &LT- DEST

e.g.
            mov         $0,     %al
    lock    cmpxchg     %edx,   some_lock

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

Machine Check Handling (AMD64)

This is meant to be a supplement to

Machine check handling on Linux by Andi Kleen.

References:

● https://www.halobates.de/

This page is under CC-BY-SA-4.0, code snippets from the Linux project(s) are under GPL-2.0-only. The above note must be preserved.

----------------------

Takeaways from Andi Kleen paper

Two types machine check

● machine check exceptions (MCEs) / uncorrectable error. It will cause interrupt w/ a special excaption handler.

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))

ia32 instructions and micro-arch for locks

back to index
PAUSE instructionSpin Loop Hint, give hint to processor that improves performance of spin-wait loops. Specifically, to transfer yield CPU to another hyperthread.
The PAUSE instruction provides a hint to the processor that the code sequence is a spin-wait loop. The processor uses this hint to avoid the memory order violation in most situations, which greatly improves processor performance.

An additional function of the PAUSE instruction is to reduce the power consumed by a processor while executing a spin loop. A processor can execute a spin-wait loop extremely quickly, causing the processor to consume a lot of power while it waits for the resource it is spinning on to become available. Inserting a pause instruction in a spin-wait loop greatly reduces the processor’s power consumption.

GCC _mm_pause() expands to __builtin_ia32_pause() then the PAUSE instruction.

This can be used to improve the spinlock performance (instead of a NOP in the busy loop).

while (__stomic_sub_fetch(&lock, 1, __ATOMIC_ACQUIRE) < 0) {
    do
        _mm_pause();
    while (__atomic_load_n(&lock, __ATOMIC_ACQUIRE) != 1);
}

via home - SHRIK3 on SHRIK3 (author: i@shrik3.com (SHRIK3))