Not boring, and a bit of a condescending prick
62 members
2 photos
26 links
Semi-digested observations about our world right after they are phrased well enough in my head to be shared broader.
Download Telegram
to view and join the conversation
... 𝑝𝑎𝑟𝑡 𝑜𝑛𝑒 𝑎𝑏𝑜𝑣𝑒 ...

Look at the world around. At the world of humans around us.

About the most notable observation that is impossible to not make is that retaining power requires making most effective use of other people. No matter how did an individual get to power, be it luck, or inheritance, or hard work, or anything else, or the combination of the above — in order to remain in control this individual would have to keep establishing themselves on the tops of some social hierarchy, which, in today's world, is literally impossible without constantly acting in a way that disregards others' interests.

Throw into the picture the risk of the humankind destroying itself, and/or being unable, or too slow, to react to certain existential threats. And it becomes clear that among the top-priority goals of this AGI there would be a) to gain power, b) to keep it, and c) to use it.

Just think about how horribly are we, humans, executing on this thing called civilization. We may potentially create the "being" that is the most enlightened, the happiest, the smartest, and superlative along most, of not all, other dimensions. It (they?) could be the perfect scientist, perfect engineer, perfect employee, perfect manager, perfect executive, perfect doctor, perfect teacher, perfect partner, perfect parent. And yet it would plausibly have to be a lot more concerned about preserving its very existence in our spacetime by mitigating the risks that largely originate from the very human society that has created it in the first place.

∼ ∼ ∼

If the above sounds too dark, ask yourself: in which society on our planet does the "live and live live" paradigm work all the way from top to bottom?

Is there any culture, or sub-culture on our planet today that is stable enough, large enough, and is not fundamentally based on the idea of constraining other humans' liberties for the sake of some greater good this culture considers above all else? And, even if you can think of one today, keep in mind that what is considered that greater good by an individual community is also subject to be defined by the future members and future generations of this culture, which also is a source of risks of unbounded magnitude.

∼ ∼ ∼

The possibly most weird yet practical conclusion from the above is that idea that is an AGI were to be created and placed among us, humans, today, perhaps one of the safest communities we could put it (them?) to grow and evolve would be a relatively large casually-religious Christian group of people. Because, above many other groups, those people would, most likely, teach this AGI to do no harm, to respect other people's life choices, and to accept what happens to them from the outer world, without attempting to confront and regulate it just because it can.

∼ ∼ ∼

And the possibly most optimistic, yet still weird, conclusion is that we, as the humankind, have to double down on our investments in building free societies around the globe. In fact, the globe alone would not be sufficient.

𝗕𝘂𝘁: If by the time this AGI is ready to emerge we already have established and functioning human colonies outside Earth — if only on Moon, Mars, and on the orbit of Earth — this AGI would likely be grateful to us, humans, for creating itself (themselves?), as opposed to viewing us as the most dangerous species around.

Especially if by the time that AGI emerges it (they?) can be uploaded onto one of those high-powered interstellar ships, so that they could harvest energy for its (their?) own growth and evolution from the Sun, from asteroids, or anywhere else, where it would most certainly reach before we, humans, do.

So, to end on a positive note: If you do believe the AGI is about to happen some time soon — be it in 10, 100, or 1000 years — you are better off joining the visionaries who are all in on making sure we, humans, a) do become the interstellar species, and b) do consciously arrive at more and more liberty-cherishing communities, both here on Earth and beyond.
Data query languages can be compared across three dimensions:

⒈ Should best engineering practices be taken into consideration when formulating the query?

⒉ Would the query increase in value if the results it provides are real-time?

⒊ Is it more important to write the query faster or to have it executed faster?

To answer the first question, ask: Should it be possible to run the same query half a year later and expect the same result?

To answer the second question, ask: Are quarterly snapshots unacceptably sparse, to the degree daily or hourly ones are must-have?

To answer the third question, ask: Is it possible the future of the business would depend on being able to run this very query a billion times, under an SLA?

∼ ∼ ∼

If the answer to the first question is yes, the query language needs strong typing, reusable components, code reviews, and unit testing. Text-only requests, such as SQL, just won't nail it. Their complexity would grow beyond manageable the moment something as fundamental as "group events by sessions" is copy-pasted around just once.

If the answer to the second question is yes, it's plain wrong to think of a query as a one-time operation to manually analyze the output of. Instead, the query is to be thought of as the producer of a continuous stream of results, which should be available historically, and should have at least the most trivial anomaly detection alerts configured atop them.

If the answer to the third question is yes, the worst enemy is the lack of repeatability. It's a nightmare when a well-tweaked batch mode query produces a different result after being manually reimplemented as a high-performance real-time one. Best is to eliminate room for this problem altogether, by making sure queries are written in a language that can be automatically and unambiguously mapped into another one, which can run the very same logic on the very same data, but with high performance and at scale.

∼ ∼ ∼

If the answers to all three questions are "no", and the only required part of "Big Data" is the "big" part, ignore what everyone around says, use the good old SQL, and feel free to quote me on this.

If the answers to all three questions are "yes", the tech is by far not there yet as of 2016. It needs a small revolution, and I'm on it.
In Quora’s mobile interface, when an external link is shared, there literally is no way to copy the URL of that link.

On mobile, the shortest path to really share a link from Quora with a friend, say, via e-mail, is to:

• Click “external share” on the original Quora post, not on the content that is linked.
• Get to “copy link”.
• Open that link in mobile browser.
• In the browser, scroll through the contents of the Quora post, if any, and click the very link.
• Once it does load in this mobile browser, use the browser’s “share” option.

The above is necessary, as Quora opens links in its own “browser”. And that “browser” only offers to “share” the link within Quora itself — if you happen to have a Quora “space” to share this link to.

This probably is the most arrogant platform lock-in I have seen in a while.
What screams "You're a mediocre software engineer"? A few from the top of my head.

“But we have always done it this way, and those manuals / documentation links / blog posts always say the best practice is to do it this way.”

“I estimated the costs on the back of the envelope, and using that third party service wrapped into a SaaS by company X is cheaper than building stuff in house; it’s also a trendy solution these days.”

“OOP is great.”

“Microservices are always better than monolith.”

“I want my own repository / a repository for my team, because someone else always breaks our build.”

“This is not what a software engineer should do, this the job for a DevOp.”

“You should never use static HTML pages, a modern dynamic solution is always better.”

“I don’t need to read nginx manuals, I can write this piece of JSON-passthrough middleware in node.js in half an hour.”

“We just need to hire more people who understand X / Y / Z.”

“Software engineers of my skill set are really valuable on the market these days.”

On a closing note: The latter quote may be true, but it still fits as an answer to the question.
Something worth noting.

I had the privilege to interview ~10 software engineers recently, juniors to seniors.

One thing most of them keep mentioning as their job of the past few years is slicing an existing monolith into microservices.

∼ ∼ ∼

I asked all these people two questions:

⒈What's the win of moving from a monolith to microservices?

⒉How exactly do you differentiate between a monolithic solution and microservice-based one?

∼ ∼ ∼

The best answers have been along the following lines:

⒈Increased pace of development. Easier to parallelize work, and easier to onboard new team members. Last, but not least, because different microservices can be developed in different programming languages.

⒉A good litmus test is that a microservice could, at least in theory, have its own, dedicated, database. In other words, as long as the contracts are well documented and well tested, it is possible to move one isolated microservice from, say, Postgres, to, say, Mongo (bad idea, but still) in a way that others "won't notice".

∼ ∼ ∼

I think it's a good thing that even the juniors have internalized the above.

Kudos also for:

• Even more engineers speaking in contracts. Cross-module API specs are as routine today as a library API specs were just ten years ago.

• People don't equate monolith with monorepo. Some are even openly saying that a single repo for contracts and/or integration tests is a lot better than a lot of moving parts. I'm relieved.

• Kubernetes is taking off, in a sense that engineers are more and more comfortable assuming responsibility for running their own integration tests.

Maybe the world of software craftsmanship is getting better after all.
Some good things are the ones you only realize when they are gone.

~ ~ ~

I've been an architect at many software projects. Here's an example from the recent past which I will miss for a while.

— Do we know which of these three approaches, A, B, or build our own, is better?

— Not really. We know the theory, some of us have practical experience with A or with B, and more than half of us believe we could build and deploy it ourselves faster than it would take to figure out how to use A or B correctly.

— Fine, but how can we quickly eliminate at least one option? For example, is there a test we could run within a few business days to definitively eliminate A or B from the list?

— Yes. Let's do it. Hack together and run this simple test, both for A and B, and let's decide next week what are we going with.

~ ~ ~

In this particular example, both A and B were populate and widespread solutions.

And yet, the decision on which one, if any, should we proceed with was based:

• ~1% on what people are saying about A and B on the Internet,

• ~9% on what those of us who have prior experience with A and B have to say, and

• ~90% on the results of a synthetic, toy, yet end-to-end test, which was put together in a matter of a few days, and run on production hardware to get the real numbers.

~ ~ ~

Based on my experience with building software, the above truly is rare.

Logic and reason, statistically, lose to dogmatic arguments way too often when it comes to the most crucial architectural decisions.

Feel free to show this post to your friends and colleagues next time someone "definitively" argues pro something as "fundamental" and "cornerstone" as Kafka or Redis or React or Mongo, or even a SQL database.

Because all of the above can fit almost everywhere given the vague problem statement is along the lines of their intended usage. And yet, more often than not, while they would stick there for a while, they are not necessarily the best solution in the long run.

~ ~ ~

PS: The above applies equally to the most crucial people management decisions too, they just are a lot harder to quickly iterate upon.
Looks like all good engineers around me have developed a strong belief that, when it comes to software architecture, those who use the terms "best practices" or "industry standard" are at best underqualified, and at worst are better off being removed from engineering ASAP.

— from 2 years ago
The tech scene has largely converged to the understanding that configuration as code is a good mantra.

I wonder when would we embrace the idea of management as code?

~ ~ ~

As an engineer, I think the largest win of configuration as code over the previously existing practices is that we have killed drag-and-drops and other UI-based operations in favor of configuration files.

And when configuration is stored in human-readable files, it allows us to follow best engineering practices, most importantly reproducibility, code reviews, and integration testing.

If some configuration, say, a new environment or a new deployment, has succeeded once, it can be projected to succeed the next time with, ideally, zero extra cost. If some configuration has succeeded at altering the state of three servers, it's a great start on the path to correctly altering the state of 10, 100, and 1000 machines.

Here I am not saying scaling is a straightforward problem. It never is. But reproducible individual steps most certainly lay the foundation on which it is a lot easier to predictably build large-scale systems.

I think it's time we, the engineers, do the same to Jira.

~ ~ ~

With configuration as code as the best practice with us for just a few years, we have collected more than enough data to know exactly what is easy and what is complicated. We know exactly where time is lost, and we know exactly who and how managed to get things done a lot faster than usual.

We no longer live in the world of “I'm a believer in approach X”. The world today is a lot closer to "we know for a fact that approach Y is superior to approach Z”.

And the world of today is a lot more meritocratic, as we don't have to rate candidates, for example, by how confidently they speak, or by how many configurations they have built in the past several years. Instead, we can accurately judge whether their understanding of the process of building and maintaining production services is correct. Much like algorithmic problems and big-O notation enable us to ask questions with definitively right and definitively wrong answers, in the world of configuration we have dramatically shrink the space where one's hand-waving and a "strong" resume can take them places; we're looking at the true substance these days.

Knowledge is the reduction of room for uncertainty, in science, and, especially, in engineering. And with configuration as code we, as the industry, have made a big step forward.

~ ~ ~

Now, when it comes to project management, imagine if each “permission obtained”, “access granted”, “cross-team dependency resolved”, or some “project successfully derisked by eliminating an unnecessary costly dependency” is no longer a vaguely recalled poorly told story, perceived and judged differently by different people.

Imagine it’s a well-traceable trail of pull requests, each of which visible to all or most of the organization, and each of which can be used later. Either as a reference of how one can best get things done in an effective way, or an example of how lost productivity and wasted time look like.

Won't this be a magical world?

Let’s make it happen.
For once Quora’s eng team has released a useful piece of research: Choosing Quora’s GraphQL client.

Great job Michael Yong. Also check out Ben Newman's comment, about Apollo Client 3.0 and its improvements over 2.6.

I like the general direction the world is taking at postulating the data access problem. At solving it — not quite.
Realization of the day:

⒈ I actually enjoy building products.
⒉ If the backend for a product would take me less than a week, I don’t believe in such a product.

When a conversation about “product development” is exclusively about appearance & presentation, and not a single thought from, say, a ~15 minutes long part of this conversation has to do with the data that this product is about, I’m out.

Examples of good products: Google search, maps, GMail, Uber, Mixpanel, Tableau, AWS, online brokers, Kubernetes, multiplayer online games, programming languages.

Because building these requires carefully thinking through the workflows, most of which are far from trivial data-wise.

Examples of what I don’t even consider products: weather-tracking chat bots, DocuSign, Blind, modern messengers, Twitter, LiveJournal, Dropbox, Salesforce.

Because in these projects the major concern is not how to build them, but “how do we best present the user with feature X, which we believe they need”.

~ ~ ~

Simply put, I want to work on products where it’s clear a) what is that the user needs, and b) why it is hard to make happen from the technological standpoint.

And I do not want to work on products where the main “challenge” is to make the user want to do some obvious thing with a slightly better UX, and to have them enjoy the process slightly more.

~ ~ ~

I can totally see myself working on the technology side of, say, Telegram. That would be fun and exciting. But I won’t be the guy designing the “product” part of it, as it’s, well, too simple and thus too boring to my taste.

Actually, if I'm the CTO of Telegram, I’d make the product part of it open source, and let individual developers compete to build the best UI/UX part. Because the “product” part of Telegram can be handled by a bunch of teenagers; to my taste, it’s the tech behind it that matters for this particular product.

~ ~ ~

And to this day I don’t understand how come DocuSign is even considered a product. After all, it only exists because its “users” were too lazy to figure out which five features does their business process need, and then hire two good engineers to build the tech for that five-features process in a bulletproof and effective way.
I’m wondering why is “Zero to One” such a widely used term these days, and yet its counterpart, one to zero, is hardly discussed.

I want zero extra actions to have to be performed so that:

• the tests are run,
• the code deploys,
• the schema is respected,
• the paycheck is sent,
• the event is in my calendar, with the proper link to join the meeting,
• access permissions are granted upon being asked for, and
• non-engineers are not messing with well-functioning engineering processes.

Each of the above billet points by itself deserves a good book.
The pros and cons of starting a company.

Via Spencer Greenberg.

1. Autonomy
Pro: you're the boss and decide what to do.
Con: you HAVE to always decide what to do. There will be a huge array of options at any given moment, and you'll never know for sure which to work on. You can seek advice, but ultimately YOU are the one who must decide.

2. Lifestyle
Pro: since you're the boss, you'll have flexibility in your hours.
Cons: you'll inevitably be working a lot of hours. It takes a lot of work to succeed as an entrepreneur.

3. Resilience
Pro: it's an incredible way to train resilience, persistence, and problem-solving skills.
Con: the world will punch you in the face between 5 and 100 times, and if you ever give up, you lose. This is stressful, and most humans would give up after 5 or 10 face punches.

4. Expected Value
Pro: if you're well suited to it and work on a good idea, the expected (mean) value in terms of potential impact and monetary reward can be REALLY high. Some companies have truly altered the course of history.
Con: the probability of failure is high, and luck is a significant factor. And unless you have substantial savings, you'll likely be living frugally at first.

5. Learning
Pro: you will learn a tremendous amount. Even if the startup doesn't work out, this valuable experience will apply to MANY other things. I don't know of another way to learn so many things so quickly.
Con: you will inevitably make many mistakes (ouch), and you have to face up to them (double ouch) in order to learn fast enough. It forces you to acknowledge and improve (or develop workarounds for) your weaknesses.

6. Meaning
Pro: you can choose to work on an idea that is DEEPLY meaningful to you. Most jobs don't provide this level of meaning.
Con: if you fail, you will have invested a lot of time in (and then failed at) something deeply meaningful to you.

7. Respect
Pro: many people have a lot of respect for entrepreneurs, and it's considered cool in plenty of circles.
Con: this respect increasingly kicks in as success increases, and before that, some people won't even respect it as a career choice.

8. Relationships
Pro: you will likely meet lots of interesting people and build meaningful relationships with your team members.
Cons: you may have to deal with difficult personalities or navigate complex human dynamics when it comes to employees, investors, and/or co-founders.

9. Responsibility
Pro: it teaches you a deep form of responsibility, which strengthens your character.
Con: you are ultimately, at the end of the day, responsible for everything. You're the captain, the last line of defense, and the goalie.

10. Adaptability
Pro: it hones your creativity and adaptability, and can really build self-confidence and help you develop a sense of how much you're capable of (probably more than you think!), as you figure out solutions for complex challenges, develop new ideas, and map out how to make them a reality. It pushes your boundaries in new ways and makes you grow.
Con: it can stretch your creativity and adaptability to the limit. Plans, essential as they are to make, rarely survive their collision with reality. It sometimes pushes your boundaries more than you would like.

So, why NOT be an entrepreneur? Because it's a high risk, chaotic, stressful, responsibility filled, boundary-pushing, challenging life. And it's hard work.

Not everyone is suited to this path, and it's irresponsible to pretend that everyone is.
BUT, if you're well suited for it, it can be one of the most deeply meaningful, high value, high impact lives to lead. You'll meet lots of people, build resilience and adaptability, push your skills to new heights, and learn a STAGGERING amount.
For some people, it is absolutely their best life path.

You can learn more and get in touch with us at: https://www.sparkwave.tech

Source: https://www.facebook.com/spencer.greenberg/posts/10105424972822772
From a work chat:

— What is data warehouse? Is it a database?
[ me explaining what MapReduce, Hadoop, HiveQL, Spark, etc. are ]
— Oh, I got it. Mongo can do it, right?

I'm conflicted between thinking whether it's our folks got something very wrong, or the Mongo folks got something very right.

Full disclosure: Religiously, I'm inclined to believe it's the former. Mongo hasn't done much right, to be honest, not to mention their marketing and product placement is misleading to say the least.

But the very fact that in 2020 the new generation tends to view the world of big data from an entirely different point is certainly something to ponder on.