Understanding Docker Internals: Building a Container Runtime in Python
https://www.reddit.com/r/programming/comments/1oiliw9/understanding_docker_internals_building_a/
submitted by /u/mraza007 (https://www.reddit.com/user/mraza007)
[link] (https://muhammadraza.me/2024/building-container-runtime-python/) [comments] (https://www.reddit.com/r/programming/comments/1oiliw9/understanding_docker_internals_building_a/)
  https://www.reddit.com/r/programming/comments/1oiliw9/understanding_docker_internals_building_a/
submitted by /u/mraza007 (https://www.reddit.com/user/mraza007)
[link] (https://muhammadraza.me/2024/building-container-runtime-python/) [comments] (https://www.reddit.com/r/programming/comments/1oiliw9/understanding_docker_internals_building_a/)
First Look at Java Valhalla: Flattening and Memory Alignment of Value Objects
https://www.reddit.com/r/programming/comments/1oix17s/first_look_at_java_valhalla_flattening_and_memory/
submitted by /u/joemwangi (https://www.reddit.com/user/joemwangi)
[link] (https://open.substack.com/pub/joemwangi985269/p/first-look-at-java-valhalla-flattening?r=2m1w1p&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false) [comments] (https://www.reddit.com/r/programming/comments/1oix17s/first_look_at_java_valhalla_flattening_and_memory/)
  https://www.reddit.com/r/programming/comments/1oix17s/first_look_at_java_valhalla_flattening_and_memory/
submitted by /u/joemwangi (https://www.reddit.com/user/joemwangi)
[link] (https://open.substack.com/pub/joemwangi985269/p/first-look-at-java-valhalla-flattening?r=2m1w1p&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false) [comments] (https://www.reddit.com/r/programming/comments/1oix17s/first_look_at_java_valhalla_flattening_and_memory/)
🧠 Exploring coding challenge platforms — which ones actually help you grow as a developer?
https://www.reddit.com/r/programming/comments/1oixvcn/exploring_coding_challenge_platforms_which_ones/
<!-- SC_OFF -->Hey folks, Over the past few weeks, I’ve been exploring various coding challenge platforms to understand how they differ — not just in problem sets, but also in how they impact real skill growth for developers. Some focus on interview-style DSA questions, others emphasize language mastery or competitive programming, and a few even encourage collaboration and discussion. I put together a short write-up summarizing what I found useful (and not so useful) across popular platforms — from LeetCode to Codeforces, HackerRank, and others. Sharing it here in case anyone’s interested in comparing experiences or adding platforms I missed:
🔗 Best Coding Challenge Platforms: LeetCode, HackerRank & More (https://sagarnikam123.github.io/posts/best-coding-challenge-platforms/) I’m curious — for those who actively use challenge sites,
👉 Which platform do you feel provides the best long-term learning value?
👉 And which ones are overrated or just “grind traps”? Would love to hear your thoughts — especially from those mentoring juniors or hiring devs who use these platforms regularly. <!-- SC_ON --> submitted by /u/sagarnikam123 (https://www.reddit.com/user/sagarnikam123)
[link] (https://sagarnikam123.github.io/posts/best-coding-challenge-platforms/) [comments] (https://www.reddit.com/r/programming/comments/1oixvcn/exploring_coding_challenge_platforms_which_ones/)
  https://www.reddit.com/r/programming/comments/1oixvcn/exploring_coding_challenge_platforms_which_ones/
<!-- SC_OFF -->Hey folks, Over the past few weeks, I’ve been exploring various coding challenge platforms to understand how they differ — not just in problem sets, but also in how they impact real skill growth for developers. Some focus on interview-style DSA questions, others emphasize language mastery or competitive programming, and a few even encourage collaboration and discussion. I put together a short write-up summarizing what I found useful (and not so useful) across popular platforms — from LeetCode to Codeforces, HackerRank, and others. Sharing it here in case anyone’s interested in comparing experiences or adding platforms I missed:
🔗 Best Coding Challenge Platforms: LeetCode, HackerRank & More (https://sagarnikam123.github.io/posts/best-coding-challenge-platforms/) I’m curious — for those who actively use challenge sites,
👉 Which platform do you feel provides the best long-term learning value?
👉 And which ones are overrated or just “grind traps”? Would love to hear your thoughts — especially from those mentoring juniors or hiring devs who use these platforms regularly. <!-- SC_ON --> submitted by /u/sagarnikam123 (https://www.reddit.com/user/sagarnikam123)
[link] (https://sagarnikam123.github.io/posts/best-coding-challenge-platforms/) [comments] (https://www.reddit.com/r/programming/comments/1oixvcn/exploring_coding_challenge_platforms_which_ones/)
How to create Object copies efficiently in Java without rebuilding them from scratch?
https://www.reddit.com/r/programming/comments/1oiyaj1/how_to_create_object_copies_efficiently_in_java/
<!-- SC_OFF -->Let's go through a beginner-friendly guide on the Prototype Design Pattern in Java: One of the most practical creational patterns when you need to create new objects by cloning existing ones instead of building them from scratch. This article covers: What the Prototype Design Pattern is (in plain English) Shallow vs Deep Copy — explained with visuals Modern Java 21 code examples (no outdated Cloneable mess) UML diagram & Sequence Diagram for better understanding Common interview questions and FAQs If you’re preparing for Java interviews, learning design patterns, or just want to level up your Java design skills, this will help a lot. Read the full article here: Prototype Design Pattern in Java With Examples (https://javatechonline.com/prototype-design-pattern-in-java-examples/) <!-- SC_ON --> submitted by /u/erdsingh24 (https://www.reddit.com/user/erdsingh24)
[link] (https://javatechonline.com/prototype-design-pattern-in-java-examples/) [comments] (https://www.reddit.com/r/programming/comments/1oiyaj1/how_to_create_object_copies_efficiently_in_java/)
  https://www.reddit.com/r/programming/comments/1oiyaj1/how_to_create_object_copies_efficiently_in_java/
<!-- SC_OFF -->Let's go through a beginner-friendly guide on the Prototype Design Pattern in Java: One of the most practical creational patterns when you need to create new objects by cloning existing ones instead of building them from scratch. This article covers: What the Prototype Design Pattern is (in plain English) Shallow vs Deep Copy — explained with visuals Modern Java 21 code examples (no outdated Cloneable mess) UML diagram & Sequence Diagram for better understanding Common interview questions and FAQs If you’re preparing for Java interviews, learning design patterns, or just want to level up your Java design skills, this will help a lot. Read the full article here: Prototype Design Pattern in Java With Examples (https://javatechonline.com/prototype-design-pattern-in-java-examples/) <!-- SC_ON --> submitted by /u/erdsingh24 (https://www.reddit.com/user/erdsingh24)
[link] (https://javatechonline.com/prototype-design-pattern-in-java-examples/) [comments] (https://www.reddit.com/r/programming/comments/1oiyaj1/how_to_create_object_copies_efficiently_in_java/)
Structuring multi-agent AI systems efficiently
https://www.reddit.com/r/programming/comments/1oiypfv/structuring_multiagent_ai_systems_efficiently/
<!-- SC_OFF -->I’m experimenting with AI agents that must work across multiple messaging apps while remembering context. Using Photon, I could prototype quickly with less boilerplate. How do you usually structure multi-agent AI systems to make them modular, maintainable, and memory-aware? Any recommended patterns or frameworks? <!-- SC_ON --> submitted by /u/Fearless-Confusion-4 (https://www.reddit.com/user/Fearless-Confusion-4)
[link] (https://photon.codes/) [comments] (https://www.reddit.com/r/programming/comments/1oiypfv/structuring_multiagent_ai_systems_efficiently/)
  https://www.reddit.com/r/programming/comments/1oiypfv/structuring_multiagent_ai_systems_efficiently/
<!-- SC_OFF -->I’m experimenting with AI agents that must work across multiple messaging apps while remembering context. Using Photon, I could prototype quickly with less boilerplate. How do you usually structure multi-agent AI systems to make them modular, maintainable, and memory-aware? Any recommended patterns or frameworks? <!-- SC_ON --> submitted by /u/Fearless-Confusion-4 (https://www.reddit.com/user/Fearless-Confusion-4)
[link] (https://photon.codes/) [comments] (https://www.reddit.com/r/programming/comments/1oiypfv/structuring_multiagent_ai_systems_efficiently/)
Web Development In… Pascal?
https://www.reddit.com/r/programming/comments/1oj0fos/web_development_in_pascal/
submitted by /u/self (https://www.reddit.com/user/self)
[link] (https://hackaday.com/2025/10/28/web-development-in-pascal/) [comments] (https://www.reddit.com/r/programming/comments/1oj0fos/web_development_in_pascal/)
  https://www.reddit.com/r/programming/comments/1oj0fos/web_development_in_pascal/
submitted by /u/self (https://www.reddit.com/user/self)
[link] (https://hackaday.com/2025/10/28/web-development-in-pascal/) [comments] (https://www.reddit.com/r/programming/comments/1oj0fos/web_development_in_pascal/)
Making Sense of Lambda Calculus 6: Recurring Problems
https://www.reddit.com/r/programming/comments/1oj0o60/making_sense_of_lambda_calculus_6_recurring/
submitted by /u/aartaka (https://www.reddit.com/user/aartaka)
[link] (https://aartaka.me/lambda-6.html) [comments] (https://www.reddit.com/r/programming/comments/1oj0o60/making_sense_of_lambda_calculus_6_recurring/)
  https://www.reddit.com/r/programming/comments/1oj0o60/making_sense_of_lambda_calculus_6_recurring/
submitted by /u/aartaka (https://www.reddit.com/user/aartaka)
[link] (https://aartaka.me/lambda-6.html) [comments] (https://www.reddit.com/r/programming/comments/1oj0o60/making_sense_of_lambda_calculus_6_recurring/)
Disasters I've seen in a microservices world, part II
https://www.reddit.com/r/programming/comments/1oj1i7i/disasters_ive_seen_in_a_microservices_world_part/
<!-- SC_OFF -->Four years ago, I wrote Disasters I've Seen in a Microservices World. I thought by now we'd have solved most of them. We didn't. We just learned to live with the chaos. The sequel is out. Four new "disasters” I've seen first-hand: #7 more services than engineers #8 the gateway to hell #9 technology sprawl #10 when the org chart becomes your architecture Does it sound familiar to you? <!-- SC_ON --> submitted by /u/joaoqalves (https://www.reddit.com/user/joaoqalves)
[link] (https://world.hey.com/joaoqalves/disasters-i-ve-seen-in-a-microservices-world-part-ii-9e6826bf) [comments] (https://www.reddit.com/r/programming/comments/1oj1i7i/disasters_ive_seen_in_a_microservices_world_part/)
  https://www.reddit.com/r/programming/comments/1oj1i7i/disasters_ive_seen_in_a_microservices_world_part/
<!-- SC_OFF -->Four years ago, I wrote Disasters I've Seen in a Microservices World. I thought by now we'd have solved most of them. We didn't. We just learned to live with the chaos. The sequel is out. Four new "disasters” I've seen first-hand: #7 more services than engineers #8 the gateway to hell #9 technology sprawl #10 when the org chart becomes your architecture Does it sound familiar to you? <!-- SC_ON --> submitted by /u/joaoqalves (https://www.reddit.com/user/joaoqalves)
[link] (https://world.hey.com/joaoqalves/disasters-i-ve-seen-in-a-microservices-world-part-ii-9e6826bf) [comments] (https://www.reddit.com/r/programming/comments/1oj1i7i/disasters_ive_seen_in_a_microservices_world_part/)
How Remote Procedure Call Works
https://www.reddit.com/r/programming/comments/1oj1o2t/how_remote_procedure_call_works/
submitted by /u/sdxyz42 (https://www.reddit.com/user/sdxyz42)
[link] (https://newsletter.systemdesign.one/p/how-rpc-works) [comments] (https://www.reddit.com/r/programming/comments/1oj1o2t/how_remote_procedure_call_works/)
  https://www.reddit.com/r/programming/comments/1oj1o2t/how_remote_procedure_call_works/
submitted by /u/sdxyz42 (https://www.reddit.com/user/sdxyz42)
[link] (https://newsletter.systemdesign.one/p/how-rpc-works) [comments] (https://www.reddit.com/r/programming/comments/1oj1o2t/how_remote_procedure_call_works/)
Tips for stroke-surviving software engineers
https://www.reddit.com/r/programming/comments/1oj1yeo/tips_for_strokesurviving_software_engineers/
submitted by /u/No-Session6643 (https://www.reddit.com/user/No-Session6643)
[link] (https://blog.j11y.io/2025-10-29_stroke_tips_for_engineers/) [comments] (https://www.reddit.com/r/programming/comments/1oj1yeo/tips_for_strokesurviving_software_engineers/)
  https://www.reddit.com/r/programming/comments/1oj1yeo/tips_for_strokesurviving_software_engineers/
submitted by /u/No-Session6643 (https://www.reddit.com/user/No-Session6643)
[link] (https://blog.j11y.io/2025-10-29_stroke_tips_for_engineers/) [comments] (https://www.reddit.com/r/programming/comments/1oj1yeo/tips_for_strokesurviving_software_engineers/)
Build your own Search Engine from Scratch in Java
https://www.reddit.com/r/programming/comments/1oj4477/build_your_own_search_engine_from_scratch_in_java/
submitted by /u/kishunkumaar (https://www.reddit.com/user/kishunkumaar)
[link] (https://www.0xkishan.com/blogs/build-your-own-search-engine-from-scratch-in-java) [comments] (https://www.reddit.com/r/programming/comments/1oj4477/build_your_own_search_engine_from_scratch_in_java/)
  https://www.reddit.com/r/programming/comments/1oj4477/build_your_own_search_engine_from_scratch_in_java/
submitted by /u/kishunkumaar (https://www.reddit.com/user/kishunkumaar)
[link] (https://www.0xkishan.com/blogs/build-your-own-search-engine-from-scratch-in-java) [comments] (https://www.reddit.com/r/programming/comments/1oj4477/build_your_own_search_engine_from_scratch_in_java/)
Kafka is fast -- I'll use Postgres
https://www.reddit.com/r/programming/comments/1oj7q6q/kafka_is_fast_ill_use_postgres/
submitted by /u/arshidwahga (https://www.reddit.com/user/arshidwahga)
[link] (https://topicpartition.io/blog/postgres-pubsub-queue-benchmarks) [comments] (https://www.reddit.com/r/programming/comments/1oj7q6q/kafka_is_fast_ill_use_postgres/)
  https://www.reddit.com/r/programming/comments/1oj7q6q/kafka_is_fast_ill_use_postgres/
submitted by /u/arshidwahga (https://www.reddit.com/user/arshidwahga)
[link] (https://topicpartition.io/blog/postgres-pubsub-queue-benchmarks) [comments] (https://www.reddit.com/r/programming/comments/1oj7q6q/kafka_is_fast_ill_use_postgres/)
From VS Code to Helix
https://www.reddit.com/r/programming/comments/1oj7xhq/from_vs_code_to_helix/
submitted by /u/Leading-Youth6865 (https://www.reddit.com/user/Leading-Youth6865)
[link] (https://ergaster.org/posts/2025/10/29-vscode-to-helix/) [comments] (https://www.reddit.com/r/programming/comments/1oj7xhq/from_vs_code_to_helix/)
  https://www.reddit.com/r/programming/comments/1oj7xhq/from_vs_code_to_helix/
submitted by /u/Leading-Youth6865 (https://www.reddit.com/user/Leading-Youth6865)
[link] (https://ergaster.org/posts/2025/10/29-vscode-to-helix/) [comments] (https://www.reddit.com/r/programming/comments/1oj7xhq/from_vs_code_to_helix/)
"The Bug Hunt" blog post pattern
https://www.reddit.com/r/programming/comments/1oj9dkh/the_bug_hunt_blog_post_pattern/
<!-- SC_OFF -->This is Chapter 8 of the book "Writing for Developers: Blogs That Get Read" (published by Manning). And here's an ever-growing collection of “Bug Hunt” blog posts https://writethat.blog/?pattern=bug%20hunt <!-- SC_ON --> submitted by /u/swdevtest (https://www.reddit.com/user/swdevtest)
[link] (https://writethatblog.substack.com/p/the-bug-hunt-blog-post-pattern) [comments] (https://www.reddit.com/r/programming/comments/1oj9dkh/the_bug_hunt_blog_post_pattern/)
  https://www.reddit.com/r/programming/comments/1oj9dkh/the_bug_hunt_blog_post_pattern/
<!-- SC_OFF -->This is Chapter 8 of the book "Writing for Developers: Blogs That Get Read" (published by Manning). And here's an ever-growing collection of “Bug Hunt” blog posts https://writethat.blog/?pattern=bug%20hunt <!-- SC_ON --> submitted by /u/swdevtest (https://www.reddit.com/user/swdevtest)
[link] (https://writethatblog.substack.com/p/the-bug-hunt-blog-post-pattern) [comments] (https://www.reddit.com/r/programming/comments/1oj9dkh/the_bug_hunt_blog_post_pattern/)
Let Us Open URL's in a Specific Browser Profile
https://www.reddit.com/r/programming/comments/1ojagkh/let_us_open_urls_in_a_specific_browser_profile/
submitted by /u/ekrubnivek (https://www.reddit.com/user/ekrubnivek)
[link] (https://kevin.burke.dev/kevin/open-urls-in-specific-browser-profile/?reddit) [comments] (https://www.reddit.com/r/programming/comments/1ojagkh/let_us_open_urls_in_a_specific_browser_profile/)
  https://www.reddit.com/r/programming/comments/1ojagkh/let_us_open_urls_in_a_specific_browser_profile/
submitted by /u/ekrubnivek (https://www.reddit.com/user/ekrubnivek)
[link] (https://kevin.burke.dev/kevin/open-urls-in-specific-browser-profile/?reddit) [comments] (https://www.reddit.com/r/programming/comments/1ojagkh/let_us_open_urls_in_a_specific_browser_profile/)
Beating Neural Networks with Batch Compression: A 3.50x Result on comma.ai’s Vector Quantization Challenge
https://www.reddit.com/r/programming/comments/1ojaus4/beating_neural_networks_with_batch_compression_a/
submitted by /u/Unusual_Midnight_523 (https://www.reddit.com/user/Unusual_Midnight_523)
[link] (https://medium.com/@YuvalLevental/beating-neural-networks-with-batch-compression-a-3-50x-result-on-comma-ais-vq-challenge-d7fb330073d2) [comments] (https://www.reddit.com/r/programming/comments/1ojaus4/beating_neural_networks_with_batch_compression_a/)
  https://www.reddit.com/r/programming/comments/1ojaus4/beating_neural_networks_with_batch_compression_a/
submitted by /u/Unusual_Midnight_523 (https://www.reddit.com/user/Unusual_Midnight_523)
[link] (https://medium.com/@YuvalLevental/beating-neural-networks-with-batch-compression-a-3-50x-result-on-comma-ais-vq-challenge-d7fb330073d2) [comments] (https://www.reddit.com/r/programming/comments/1ojaus4/beating_neural_networks_with_batch_compression_a/)
Azure down
https://www.reddit.com/r/programming/comments/1ojbiss/azure_down/
submitted by /u/RR_2025 (https://www.reddit.com/user/RR_2025)
[link] (https://azure.status.microsoft/en-us/status) [comments] (https://www.reddit.com/r/programming/comments/1ojbiss/azure_down/)
  https://www.reddit.com/r/programming/comments/1ojbiss/azure_down/
submitted by /u/RR_2025 (https://www.reddit.com/user/RR_2025)
[link] (https://azure.status.microsoft/en-us/status) [comments] (https://www.reddit.com/r/programming/comments/1ojbiss/azure_down/)
How We Continually Deliver Software
https://www.reddit.com/r/programming/comments/1ojbos8/how_we_continually_deliver_software/
submitted by /u/paltman94 (https://www.reddit.com/user/paltman94)
[link] (https://wedgworth.dev/how-we-continually-deliver-software/) [comments] (https://www.reddit.com/r/programming/comments/1ojbos8/how_we_continually_deliver_software/)
  https://www.reddit.com/r/programming/comments/1ojbos8/how_we_continually_deliver_software/
submitted by /u/paltman94 (https://www.reddit.com/user/paltman94)
[link] (https://wedgworth.dev/how-we-continually-deliver-software/) [comments] (https://www.reddit.com/r/programming/comments/1ojbos8/how_we_continually_deliver_software/)
Vi /Vim Editor : Practical commands every developer, sysadmin, and DevOps engineer should know.
https://www.reddit.com/r/programming/comments/1ojbu0m/vi_vim_editor_practical_commands_every_developer/
<!-- SC_OFF -->I have put together a simple guide to vi commands that actually helped me all these years when editing configs or scripts on Linux.
Short, practical, and focused on real examples. Let me know if I have missed some..would love to take feedbacks and make it an exhaustive list! Read it here (https://medium.com/stackademic/the-80-20-guide-to-vi-editor-20-of-commands-that-do-80-of-the-work-ff1ce320f461?sk=b61c38fdc6a69ae5ba7400390964934b) <!-- SC_ON --> submitted by /u/sshetty03 (https://www.reddit.com/user/sshetty03)
[link] (https://medium.com/stackademic/the-80-20-guide-to-vi-editor-20-of-commands-that-do-80-of-the-work-ff1ce320f461?sk=b61c38fdc6a69ae5ba7400390964934b) [comments] (https://www.reddit.com/r/programming/comments/1ojbu0m/vi_vim_editor_practical_commands_every_developer/)
  https://www.reddit.com/r/programming/comments/1ojbu0m/vi_vim_editor_practical_commands_every_developer/
<!-- SC_OFF -->I have put together a simple guide to vi commands that actually helped me all these years when editing configs or scripts on Linux.
Short, practical, and focused on real examples. Let me know if I have missed some..would love to take feedbacks and make it an exhaustive list! Read it here (https://medium.com/stackademic/the-80-20-guide-to-vi-editor-20-of-commands-that-do-80-of-the-work-ff1ce320f461?sk=b61c38fdc6a69ae5ba7400390964934b) <!-- SC_ON --> submitted by /u/sshetty03 (https://www.reddit.com/user/sshetty03)
[link] (https://medium.com/stackademic/the-80-20-guide-to-vi-editor-20-of-commands-that-do-80-of-the-work-ff1ce320f461?sk=b61c38fdc6a69ae5ba7400390964934b) [comments] (https://www.reddit.com/r/programming/comments/1ojbu0m/vi_vim_editor_practical_commands_every_developer/)
Educational Benchmark: 100 Million Records with Mobile Logic Compression (Python + SQLite + Zlib)
https://www.reddit.com/r/programming/comments/1ojc99d/educational_benchmark_100_million_records_with/
<!-- SC_OFF -->Introduction This is an educational and exploratory experiment on how Python can handle large volumes of data by applying logical and semantic compression, a concept I called LSC (Logical Semantic Compression). The proposal was to generate 100 million structured records and store them in compressed blocks, using only Python, SQLite and Zlib — without parallelism and without high-performance external libraries. ⚙️ Environment Configuration Device: Android (via Termux) Language: Python 3 Database: SQLite Compression: zlib Mode: Singlecore Total records: 100,000,000 Batch: 1,000 records per chunk Periodic commits: every 3 chunks 🧩 Logical Structure Each record generated follows a simple semantic pattern: { "id": i, "title": f"Book {i}", "author": "random letter string", "year": number between 1950 and 2024, "category": "Romance/Science/History" } These records are grouped into chunks and, before being stored in the database, they are converted into JSON and compressed with zlib. Each block represents a “logical package” — a central concept in LSC. ⚙️ Main Excerpt from the Code json_bytes = json.dumps(batch, separators=(',', ':')).encode() comp_blob = zlib.compress(json_bytes, ZLIB_LEVEL) cur.execute( "INSERT INTO chunks (start_id, end_id, blob, count) VALUES (?, ?, ?, ?)", (i - BATCH_SIZE + 1, i, sqlite3.Binary(comp_blob), len(batch)) ) The code executes: Semantic generation of records JSON Serialization Logic compression (Zlib) Writing to SQLite 🚀 Benchmark Results Result Metric 📊 100,000,000 records generated 🧩 Chunks processed 100,000 📦 Compressed size ~2 GB 📤 Uncompressed size ~10 GB ⚙️ Compression ratio ~20% ⏱️ Total time ~50 seconds (approx.) ⚡ Average speed ~200,000 records/s 🔸 Singlecore Mode (CPU-bound) 🔬 Observations Even though it was run on a smartphone, the result was surprisingly stable. The compression rate remained close to 20%, with minimal variation between blocks. This demonstrates that, with a good logical data structure, it is possible to achieve considerable efficiency without resorting to parallelism or optimizations in C/C++. 🧠 About LSC LSC (Logical Semantic Compression) is not a library, but an idea: Compress data based on its logical structure and semantic repetition, not just in the raw bytes. Thus, each block carries not only information, but also relationships and coherence between records. Compression becomes a reflection of the meaning of the data — not just its size. 🎓 Conclusion Even running in singlecore mode and with simple configurations, Python showed that it is possible to handle 100 million structured records, maintaining consistent compression and low fragmentation. 🔍 This experiment reinforces the idea that the logical organization of data can be as powerful as technical optimization. <!-- SC_ON --> submitted by /u/MajorPistola (https://www.reddit.com/user/MajorPistola)
[link] (https://www.reddit.com/r/datascience/comments/1oj8ufa/educational_benchmark_100_million_records_with/?share_id=9ZYXguvpWPZkl3y91a5Xg&utm_content=1&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1) [comments] (https://www.reddit.com/r/programming/comments/1ojc99d/educational_benchmark_100_million_records_with/)
  https://www.reddit.com/r/programming/comments/1ojc99d/educational_benchmark_100_million_records_with/
<!-- SC_OFF -->Introduction This is an educational and exploratory experiment on how Python can handle large volumes of data by applying logical and semantic compression, a concept I called LSC (Logical Semantic Compression). The proposal was to generate 100 million structured records and store them in compressed blocks, using only Python, SQLite and Zlib — without parallelism and without high-performance external libraries. ⚙️ Environment Configuration Device: Android (via Termux) Language: Python 3 Database: SQLite Compression: zlib Mode: Singlecore Total records: 100,000,000 Batch: 1,000 records per chunk Periodic commits: every 3 chunks 🧩 Logical Structure Each record generated follows a simple semantic pattern: { "id": i, "title": f"Book {i}", "author": "random letter string", "year": number between 1950 and 2024, "category": "Romance/Science/History" } These records are grouped into chunks and, before being stored in the database, they are converted into JSON and compressed with zlib. Each block represents a “logical package” — a central concept in LSC. ⚙️ Main Excerpt from the Code json_bytes = json.dumps(batch, separators=(',', ':')).encode() comp_blob = zlib.compress(json_bytes, ZLIB_LEVEL) cur.execute( "INSERT INTO chunks (start_id, end_id, blob, count) VALUES (?, ?, ?, ?)", (i - BATCH_SIZE + 1, i, sqlite3.Binary(comp_blob), len(batch)) ) The code executes: Semantic generation of records JSON Serialization Logic compression (Zlib) Writing to SQLite 🚀 Benchmark Results Result Metric 📊 100,000,000 records generated 🧩 Chunks processed 100,000 📦 Compressed size ~2 GB 📤 Uncompressed size ~10 GB ⚙️ Compression ratio ~20% ⏱️ Total time ~50 seconds (approx.) ⚡ Average speed ~200,000 records/s 🔸 Singlecore Mode (CPU-bound) 🔬 Observations Even though it was run on a smartphone, the result was surprisingly stable. The compression rate remained close to 20%, with minimal variation between blocks. This demonstrates that, with a good logical data structure, it is possible to achieve considerable efficiency without resorting to parallelism or optimizations in C/C++. 🧠 About LSC LSC (Logical Semantic Compression) is not a library, but an idea: Compress data based on its logical structure and semantic repetition, not just in the raw bytes. Thus, each block carries not only information, but also relationships and coherence between records. Compression becomes a reflection of the meaning of the data — not just its size. 🎓 Conclusion Even running in singlecore mode and with simple configurations, Python showed that it is possible to handle 100 million structured records, maintaining consistent compression and low fragmentation. 🔍 This experiment reinforces the idea that the logical organization of data can be as powerful as technical optimization. <!-- SC_ON --> submitted by /u/MajorPistola (https://www.reddit.com/user/MajorPistola)
[link] (https://www.reddit.com/r/datascience/comments/1oj8ufa/educational_benchmark_100_million_records_with/?share_id=9ZYXguvpWPZkl3y91a5Xg&utm_content=1&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1) [comments] (https://www.reddit.com/r/programming/comments/1ojc99d/educational_benchmark_100_million_records_with/)