But because I am terrified by the prospect of decoupling the actual wealth, that maps to something tangible, and the "numbers on the bank accounts", which are what the modern "economy" increasingly is about.
Thomas Cook, the British travel agency, is no more.
I may be off in numbers, but it looks like over $0.5B will be taken off the UK budget — read: will be paid by UK taxpayers — in order to get the "poor, lost, abandoned" tourists back home.
There is something fundamentally wrong here.
Everyone knew Thomas Cook struggles for the past several years. It was common knowledge that bankruptcy is a likely scenario.
And yet the, presumably poorer, citizens of the UK — who were not on vacation — are paying for the relatively peaceful endings of vacations of the, presumably richer, citizens of the UK — who decided to take vacations with Thomas Cook nonetheless.
∼ ∼ ∼
This looks a lot like The Big Short playbook.
Even if you know the market is a huge bubble that is about to
collapse, you also know "the government" will eventually be "on your side" — i.e., you know that the [other] taxpayers' money will be used to pay for your "stupidity".
In other words, the current socialistic governments system supports the incentives of acting in a "stupid" way, even if you are the opposite of the "stupid" actor.
Such as taking another house or condo loan in 2008, even knowing exactly what is about to happen.
Or such as booking a cheaper tour with Thomas Cook, knowing well it's on the verge of bankruptcy.
∼ ∼ ∼
I don't know what the solution to the above problem may be.
Maybe, there is no problem, after all. On occasions like this all taxpayers will have to pay a few, or a few dozen, bucks, and it will happen once or twice a year. Maybe it really is not a big deal. Especially given that we are consciously paying a lot more in taxes, knowing with confidence that those funds are not being spent well.
But, fundamentally, the incentives scheme has to change.
∼ ∼ ∼
In this particular case, for example, the British government could have published a memo, a few years back, stating that everyone traveling with Thomas Cook must also purchase the respective state-approved insurance package.
So that it's not every taxpayer who will end up paying up after the collapse, but every Thomas Cook traveler from the past few years.
And then the UK could state openly that they will not spend a single penny towards helping those who decided to ignore this warning, and travel with Thomas Cook uninsured. Because they have consciously assumed this risk onto themselves.
∼ ∼ ∼
I know I'm daydreaming here. But this topic of being more conscious about what exactly are we paying taxes for is growing on me.
Much like we seem to care more about the environment and about minorities' rights these days, we might well begin to be more conscious about the magnitude of incompetence in how do our governments use our, taxpayers', money.
∼ ∼ ∼
And then maybe, maybe, one day we will get to being conscious about which technologies do we use. Because a poorly designed Web framework, or a poorly designed machine learning library may well be contributing more to the CO₂ emission than gasoline-powered cars or international flights.
So that we will be able to push for a new standard, that would render PHP, most of Python, and most of JavaScript dangerous and obsolete.
One step at a time. Environment and minorities' right first. Then let's keep an eye on how is our money spent. And then let's launch a crusade against bad programmers who burn billions of kilowatt-hours on the activities that are, at best, useless, and, at worst, detrimental to the future of our civilization.
I may be off in numbers, but it looks like over $0.5B will be taken off the UK budget — read: will be paid by UK taxpayers — in order to get the "poor, lost, abandoned" tourists back home.
There is something fundamentally wrong here.
Everyone knew Thomas Cook struggles for the past several years. It was common knowledge that bankruptcy is a likely scenario.
And yet the, presumably poorer, citizens of the UK — who were not on vacation — are paying for the relatively peaceful endings of vacations of the, presumably richer, citizens of the UK — who decided to take vacations with Thomas Cook nonetheless.
∼ ∼ ∼
This looks a lot like The Big Short playbook.
Even if you know the market is a huge bubble that is about to
collapse, you also know "the government" will eventually be "on your side" — i.e., you know that the [other] taxpayers' money will be used to pay for your "stupidity".
In other words, the current socialistic governments system supports the incentives of acting in a "stupid" way, even if you are the opposite of the "stupid" actor.
Such as taking another house or condo loan in 2008, even knowing exactly what is about to happen.
Or such as booking a cheaper tour with Thomas Cook, knowing well it's on the verge of bankruptcy.
∼ ∼ ∼
I don't know what the solution to the above problem may be.
Maybe, there is no problem, after all. On occasions like this all taxpayers will have to pay a few, or a few dozen, bucks, and it will happen once or twice a year. Maybe it really is not a big deal. Especially given that we are consciously paying a lot more in taxes, knowing with confidence that those funds are not being spent well.
But, fundamentally, the incentives scheme has to change.
∼ ∼ ∼
In this particular case, for example, the British government could have published a memo, a few years back, stating that everyone traveling with Thomas Cook must also purchase the respective state-approved insurance package.
So that it's not every taxpayer who will end up paying up after the collapse, but every Thomas Cook traveler from the past few years.
And then the UK could state openly that they will not spend a single penny towards helping those who decided to ignore this warning, and travel with Thomas Cook uninsured. Because they have consciously assumed this risk onto themselves.
∼ ∼ ∼
I know I'm daydreaming here. But this topic of being more conscious about what exactly are we paying taxes for is growing on me.
Much like we seem to care more about the environment and about minorities' rights these days, we might well begin to be more conscious about the magnitude of incompetence in how do our governments use our, taxpayers', money.
∼ ∼ ∼
And then maybe, maybe, one day we will get to being conscious about which technologies do we use. Because a poorly designed Web framework, or a poorly designed machine learning library may well be contributing more to the CO₂ emission than gasoline-powered cars or international flights.
So that we will be able to push for a new standard, that would render PHP, most of Python, and most of JavaScript dangerous and obsolete.
One step at a time. Environment and minorities' right first. Then let's keep an eye on how is our money spent. And then let's launch a crusade against bad programmers who burn billions of kilowatt-hours on the activities that are, at best, useless, and, at worst, detrimental to the future of our civilization.
Executive Decisions
Why are corporations so slow at making executive decisions?
At any time a corporation always has several big decisions to make. Yes, some may be postponed, but generally postponing a decision is a lost opportunity: each decision should ultimately be evaluated, and either dismissed, or translated into an execution plan.
The execution itself may not begin tomorrow, and the plan may well be "we look at this again next quarter, after receiving the results of that project and that experiment and that hiring event". But that's already the execution track, not the "we're thinking about it" one.
Given we know those decision have to be made, why does making them often take weeks and quarters, not hours and days?
∼ ∼ ∼
This slowness may have to do with the cost of error, as perceived by the actor and the environment they operate in.
Consider an executive who can directly influence five decisions.
Each of those decisions can result in a gain for the company, that is an order of magnitude larger than this executive's compensation, or a loss for the company, that is an order of magnitude larger than this executive's prospective lifetime earnings.
On the one hand most opportunities, that are not discounted right away, can turn out profitable; in fact if no one from the top management team communicates a solid "over my dead body" message, the expected value of making the decision to pursue such an opportunity is likely net positive.
On the other hand, an executive who has made a cash-negative decision will be remembered as someone who has made this cash-negative decision. For the rest of their life.
∼ ∼ ∼
Now, company-wide, from the game-theoretic perspective, it all boils down to risk assessent and to constraints management.
After all, if the company can afford to pursue all five opportunities, and the expected positive outcome of each one of them is at least 4x the expected negative result of each of them, then, heck, the "default" base strategy should be to say "yes" to all five.
Also, when it comes to making decisions of this magnitude, whether the outcome is positive or negative will often only be seen much later down the road. Say, you decide that you need your own datacenter, or another engineering office, or to make a company-wide push to some new technology. It may turn out great in the long run, or it may be a disaster, but it will take years and years to see this outcome clearly.
∼ ∼ ∼
Still, I have not seen many executives who would eagerly say "yes" to all five.
They wait and wait and wait.
And a possible contributing factor could be this cost of error.
Simply put, when the executives operate in a hostile environment — where they have enemies who would do their best to get those executives fired over a misstep, or where their job prospects after being known for making a bad decision are bleak — they would indeed hesitate to move forward. Because it's too dangerous for their own career.
At the same time, the culture that embraces failures and experiences would be the culture where a) multi-billion dollar mistakes are made daily, at different places, by different people, b) and yet, the total amount of expertise and knowledge would be growing a lot faster in this culture than in the risk-averse one.
∼ ∼ ∼
So, maybe, I should rethink my views on the Silicon Valley.
Seemingly weird decisions that cost investors millions and millions of dollars are made by clearly incompletent people on a daily basis there. But that's the form everyone is used to, and it's this form that I am to allergic to. The substance is that the executives in the Bay Area a) are more experienced, and b) have access to more resources.
Seen in this light, the substance does trump the form here. And the simple idea of embracing failures, along with easy access to capital, may well be what the Silicon Valley owes its success to.
Why are corporations so slow at making executive decisions?
At any time a corporation always has several big decisions to make. Yes, some may be postponed, but generally postponing a decision is a lost opportunity: each decision should ultimately be evaluated, and either dismissed, or translated into an execution plan.
The execution itself may not begin tomorrow, and the plan may well be "we look at this again next quarter, after receiving the results of that project and that experiment and that hiring event". But that's already the execution track, not the "we're thinking about it" one.
Given we know those decision have to be made, why does making them often take weeks and quarters, not hours and days?
∼ ∼ ∼
This slowness may have to do with the cost of error, as perceived by the actor and the environment they operate in.
Consider an executive who can directly influence five decisions.
Each of those decisions can result in a gain for the company, that is an order of magnitude larger than this executive's compensation, or a loss for the company, that is an order of magnitude larger than this executive's prospective lifetime earnings.
On the one hand most opportunities, that are not discounted right away, can turn out profitable; in fact if no one from the top management team communicates a solid "over my dead body" message, the expected value of making the decision to pursue such an opportunity is likely net positive.
On the other hand, an executive who has made a cash-negative decision will be remembered as someone who has made this cash-negative decision. For the rest of their life.
∼ ∼ ∼
Now, company-wide, from the game-theoretic perspective, it all boils down to risk assessent and to constraints management.
After all, if the company can afford to pursue all five opportunities, and the expected positive outcome of each one of them is at least 4x the expected negative result of each of them, then, heck, the "default" base strategy should be to say "yes" to all five.
Also, when it comes to making decisions of this magnitude, whether the outcome is positive or negative will often only be seen much later down the road. Say, you decide that you need your own datacenter, or another engineering office, or to make a company-wide push to some new technology. It may turn out great in the long run, or it may be a disaster, but it will take years and years to see this outcome clearly.
∼ ∼ ∼
Still, I have not seen many executives who would eagerly say "yes" to all five.
They wait and wait and wait.
And a possible contributing factor could be this cost of error.
Simply put, when the executives operate in a hostile environment — where they have enemies who would do their best to get those executives fired over a misstep, or where their job prospects after being known for making a bad decision are bleak — they would indeed hesitate to move forward. Because it's too dangerous for their own career.
At the same time, the culture that embraces failures and experiences would be the culture where a) multi-billion dollar mistakes are made daily, at different places, by different people, b) and yet, the total amount of expertise and knowledge would be growing a lot faster in this culture than in the risk-averse one.
∼ ∼ ∼
So, maybe, I should rethink my views on the Silicon Valley.
Seemingly weird decisions that cost investors millions and millions of dollars are made by clearly incompletent people on a daily basis there. But that's the form everyone is used to, and it's this form that I am to allergic to. The substance is that the executives in the Bay Area a) are more experienced, and b) have access to more resources.
Seen in this light, the substance does trump the form here. And the simple idea of embracing failures, along with easy access to capital, may well be what the Silicon Valley owes its success to.
Science is the way to expand our knowledge about the laws of the Universe — physical and abstract — by means of reason and experiment.
Business is the process of broadening our knowledge about what is the customer ready to pay for — by means of launching products and analyzing the feedback.
Engineering is the art of continuously shaping our knowledge about where does the boundary of what can and what can not be built today lie — by means of pushing technologies to their limits and tacking the emerging bottlenecks one by one.
∼ ∼ ∼
In all three the key is external validation.
It is impossible for one to be delusional over an extended period of time, because an external entity would prove them wrong.
In the case of science this external entity is nature. In the case of business it's the customer.
In engineering it's when the product one used to believe is impossible to build does materialize. Or when too much resources have been sacrificed to make it safe to proclaim, beyond reasonable doubt, that certain product is impossible, or at least implausible, to be built with today's technology.
∼ ∼ ∼
A good life, or, I would say, at least a good professional life, maxes out on all three of these dimensions.
One has to simultaneously:
⒈ Get to broaden their understanding of certain first principles of our Universe,
⒉ Routinely confirm that what they are doing is what others are willing to pay for, and
⒊ Work on building something that is challenging enough, so that quite a few people around seriously believe it is impossible.
In a way, it's a shame, a misfortune, and a curse of most humans, who are or were ever alive, that they are or were ultimately forced to settle to scoring way under 3.0 by the above metric.
∼ ∼ ∼
The order of priorities of the (1), (2), and (3) above may change over time.
For instance, up until some thirty years old, I valued (3) highly, respected (1), and did not care much about (2). Today (2) has grown on me substantially, and (3) is, philosophically speaking, not much different important from (1) in my book.
Still, at no point in my life I was content to dedicate myself to doing something that had at least one of (1), (2), or (3) missing. If it's "too easy" on either of the three, it's really not worth more than a few dozen hours.
Now, the real question, of both the professional life and of life itself, is what percentage of it should we spent in those "worth a few dozen hours max" periods vs. in the "this is what I should be doing now" ones.
∼ ∼ ∼
Speaking in data science terms, what is the mode, the mean, and some p90 and p99 of that metric defined above, on the scale from 0.0 to 3.0?
Actually, paraphrasing a well-known saying, one could say:
Tell me the weights you assign to (1), (2), (3), and plot me the probability density function of your realistic expectation of in which score ranges would you be spending the next ten years of your life — and I will tell you who you are.
Business is the process of broadening our knowledge about what is the customer ready to pay for — by means of launching products and analyzing the feedback.
Engineering is the art of continuously shaping our knowledge about where does the boundary of what can and what can not be built today lie — by means of pushing technologies to their limits and tacking the emerging bottlenecks one by one.
∼ ∼ ∼
In all three the key is external validation.
It is impossible for one to be delusional over an extended period of time, because an external entity would prove them wrong.
In the case of science this external entity is nature. In the case of business it's the customer.
In engineering it's when the product one used to believe is impossible to build does materialize. Or when too much resources have been sacrificed to make it safe to proclaim, beyond reasonable doubt, that certain product is impossible, or at least implausible, to be built with today's technology.
∼ ∼ ∼
A good life, or, I would say, at least a good professional life, maxes out on all three of these dimensions.
One has to simultaneously:
⒈ Get to broaden their understanding of certain first principles of our Universe,
⒉ Routinely confirm that what they are doing is what others are willing to pay for, and
⒊ Work on building something that is challenging enough, so that quite a few people around seriously believe it is impossible.
In a way, it's a shame, a misfortune, and a curse of most humans, who are or were ever alive, that they are or were ultimately forced to settle to scoring way under 3.0 by the above metric.
∼ ∼ ∼
The order of priorities of the (1), (2), and (3) above may change over time.
For instance, up until some thirty years old, I valued (3) highly, respected (1), and did not care much about (2). Today (2) has grown on me substantially, and (3) is, philosophically speaking, not much different important from (1) in my book.
Still, at no point in my life I was content to dedicate myself to doing something that had at least one of (1), (2), or (3) missing. If it's "too easy" on either of the three, it's really not worth more than a few dozen hours.
Now, the real question, of both the professional life and of life itself, is what percentage of it should we spent in those "worth a few dozen hours max" periods vs. in the "this is what I should be doing now" ones.
∼ ∼ ∼
Speaking in data science terms, what is the mode, the mean, and some p90 and p99 of that metric defined above, on the scale from 0.0 to 3.0?
Actually, paraphrasing a well-known saying, one could say:
Tell me the weights you assign to (1), (2), (3), and plot me the probability density function of your realistic expectation of in which score ranges would you be spending the next ten years of your life — and I will tell you who you are.
Stumbled upon this amazing read: https://patrickcollison.com/fast
This, to me, is the major cause of modern social problems.
We are excessively focused on the "social sphere", and are completely ignoring the fact that the greater good for the society comes not from the "right" pronouns and the "best" antidepressants, but from the favorable living conditions we have built for ourselves, by ourselves.
And if we consciously de-prioritize execution on making those conditions better, choosing instead to focus on how to not get anyone offended, directly (no health insurance), or indirectly (speaking in "problematic" ways), I see no good happening to us in the long run.
Simply put, it's okay to my taste if a project such as the Eiffel Tower costs 50% more and is some 20% delayed, as long as those who are working on and around it get decent healthcare, maternity leave, vacation, etc.
∼ ∼ ∼
But it is absolutely not okay when our perception of reality gets distorted to the degree where an executive who can build the Eiffel Tower is mocked and displaced, and their place is taken by an "executive" who "builds" something using 10x times and 10x money.
Which is pretty much the state of the art today. Just think of modern software containerization budgets. Or of how slowly does our email open these days.
The Eiffel Tower. The Eiffel Tower was built in 2 years and 2 months; that is, in 793 days. When completed in 1889, it became the tallest building in the world, a record it held for more than 40 years. It cost about $40 million in 2019 dollars.Of course, under the tweet, where Patrick shares the link, there is more than one person replying along the lines of: think how bad were the workers treated back then, and how little were they paid.
. . .
San Francisco proposed a new bus lane on Van Ness in 2001. Its opening was recently delayed to 2020, yielding a project duration of around 7,000 days. “The project has been delayed due to an increase of wet weather since the project started,” said Paul Rose, a San Francisco Municipal Transportation Agency spokesperson. The project will cost $189 million, i.e. $60,000 per meter. The Alaska Highway, mentioned above, constructed across remote tundra, cost $793 per meter (in 2019 dollars).
This, to me, is the major cause of modern social problems.
We are excessively focused on the "social sphere", and are completely ignoring the fact that the greater good for the society comes not from the "right" pronouns and the "best" antidepressants, but from the favorable living conditions we have built for ourselves, by ourselves.
And if we consciously de-prioritize execution on making those conditions better, choosing instead to focus on how to not get anyone offended, directly (no health insurance), or indirectly (speaking in "problematic" ways), I see no good happening to us in the long run.
Simply put, it's okay to my taste if a project such as the Eiffel Tower costs 50% more and is some 20% delayed, as long as those who are working on and around it get decent healthcare, maternity leave, vacation, etc.
∼ ∼ ∼
But it is absolutely not okay when our perception of reality gets distorted to the degree where an executive who can build the Eiffel Tower is mocked and displaced, and their place is taken by an "executive" who "builds" something using 10x times and 10x money.
Which is pretty much the state of the art today. Just think of modern software containerization budgets. Or of how slowly does our email open these days.
Twitter
Patrick Collison
My new favorite example of infrastructure cost disease: construction of the Eiffel Tower cost $40 million *in 2019 dollars*. (Also, it took 2 years and 2 months to build.)
An essential trait on the way to becoming an entrepreneur is the ability to differentiate between what is fun to do and what should be done.
In this paradigm, entrepreneurship is nothing but the aggressive and persistent exploration of the boundaries and shortcuts of the world by means of scientific trial and error.
The above is impossible to be learned from the most famous entrepreneurs. Their pictures of the future just happened to be better aligned with what the future was about to bring.
In a way, those whom we know as the most successful entrepreneurs didn't become entrepreneurs; they simply discovered entrepreneurship in themselves, as a side effect of helping the world to make the right things happen.
Bad news: Reading success stories is largely useless.
Good news: At the same time the zero-to-one skill is purely execution, which is largely an orthogonal one.
Conclusion: Don't try to be an entrepreneur, instead master execution, and keep an eye out for the moment the world appears to favor your vision of the future over the alternatives.
From 4 years ago.
In this paradigm, entrepreneurship is nothing but the aggressive and persistent exploration of the boundaries and shortcuts of the world by means of scientific trial and error.
The above is impossible to be learned from the most famous entrepreneurs. Their pictures of the future just happened to be better aligned with what the future was about to bring.
In a way, those whom we know as the most successful entrepreneurs didn't become entrepreneurs; they simply discovered entrepreneurship in themselves, as a side effect of helping the world to make the right things happen.
Bad news: Reading success stories is largely useless.
Good news: At the same time the zero-to-one skill is purely execution, which is largely an orthogonal one.
Conclusion: Don't try to be an entrepreneur, instead master execution, and keep an eye out for the moment the world appears to favor your vision of the future over the alternatives.
From 4 years ago.
Often times, the simple and true answer to the "why are the things getting worse" is simple: money.
Why is Travis migration process so cumbersome and ineffective, while it worked okay for us for a long time?
It was not without issues, and we had to configure and re-configure it a number of times. But it worked. Until recently.
Why are the things getting worse then?
Because Travis wants three-digit $$$ per month, right next to saying it
Look, folks, I'm not greedy. But for my usage profile, it's three-digit cents of AWS machines per month; a lot cheaper if self-hosted.
I embrace capitalism, and may even pay. But this profit margin just feels wrong.
∼ ∼ ∼
From an individualistic and liberty-first point of view, I am, of course, not going to say Travis owes me, or anyone, anything.
And it is entirely plausible that after the team has conducted market research, enough evidence points to their $$$ number being the right price point.
It is also entirely plausible to assume the opportunity window has now closed. It is simply not worth it to enter this market by building a new CI tool, which would focus on open source, and which would be both good enough and inexpensive. First, because free tools still do exist on the market (Semaphore is our backup for a long time, and it's good), and, second, because, well, building such a tool would be an investment which has to ultimately pay off — at which point it becomes unclear if a lower price point justifies the risks.
Guess I am just sad the open source community is proving to be not as eager to push for freedoms than what it used to be. Or what I used to believe it truly is.
∼ ∼ ∼
On a second thought, it does open a bit of an opportunity window — for the brands that do want to prove they are open-source-friendly. Canonical comes to mind first.
GitHub, which is now Microsoft, by the way, may well decide to step up here too. Or a decentralized-first company, such as Urbit, might look into launching its own spread-out container-based CI for open source. I, for one, would be happy to dedicate ~0.1% of my CPUs to running the tests of someone else's open source, assuming that as long as I don't need more than an hour every eight days I would get 200% of someone else's CPUs too. Impossible to be delivered in this very setup for most projects these days, but the idea is rather clear. And it's a lot better of an idea than the "proof of work" one, which warms the planet for the sole purpose of making sure a non-governments-controlled tokens can change hands securely and safely.
But even this, second, thought though is just a second-order proof of how corporation- and conglomerate-centric is our world becoming these days. GitHub is just a brand and a user acquisition channel for Microsoft. And hosting git repositories is just one of many features that made us attracted to some brand in the first place — which ultimately gets merged into larger and larger brands that know how to make money from us all at the end of the day.
And, fundamentally, a green tick next to a commit is to a developer is what an animated shit emoji is to a regular user. People get attached to, well, their respective poisons, and those who first saw and then orchestrated those opportunities to get people attached to them benefit from those people financially.
The loop is closing in.
∼ ∼ ∼
Not complaining. It is what it is.
And I can configure Jenkins on a Hetzner box if and when we feel like the time has come. Moving off GitHub to a self-hosted git server with issue tracking and code review support is also a trivial task these days.
Just sad to witness how this dream of open [source] world vanishes as we speak.
Why is Travis migration process so cumbersome and ineffective, while it worked okay for us for a long time?
It was not without issues, and we had to configure and re-configure it a number of times. But it worked. Until recently.
Why are the things getting worse then?
Because Travis wants three-digit $$$ per month, right next to saying it
<3-s open source.Look, folks, I'm not greedy. But for my usage profile, it's three-digit cents of AWS machines per month; a lot cheaper if self-hosted.
I embrace capitalism, and may even pay. But this profit margin just feels wrong.
∼ ∼ ∼
From an individualistic and liberty-first point of view, I am, of course, not going to say Travis owes me, or anyone, anything.
And it is entirely plausible that after the team has conducted market research, enough evidence points to their $$$ number being the right price point.
It is also entirely plausible to assume the opportunity window has now closed. It is simply not worth it to enter this market by building a new CI tool, which would focus on open source, and which would be both good enough and inexpensive. First, because free tools still do exist on the market (Semaphore is our backup for a long time, and it's good), and, second, because, well, building such a tool would be an investment which has to ultimately pay off — at which point it becomes unclear if a lower price point justifies the risks.
Guess I am just sad the open source community is proving to be not as eager to push for freedoms than what it used to be. Or what I used to believe it truly is.
∼ ∼ ∼
On a second thought, it does open a bit of an opportunity window — for the brands that do want to prove they are open-source-friendly. Canonical comes to mind first.
GitHub, which is now Microsoft, by the way, may well decide to step up here too. Or a decentralized-first company, such as Urbit, might look into launching its own spread-out container-based CI for open source. I, for one, would be happy to dedicate ~0.1% of my CPUs to running the tests of someone else's open source, assuming that as long as I don't need more than an hour every eight days I would get 200% of someone else's CPUs too. Impossible to be delivered in this very setup for most projects these days, but the idea is rather clear. And it's a lot better of an idea than the "proof of work" one, which warms the planet for the sole purpose of making sure a non-governments-controlled tokens can change hands securely and safely.
But even this, second, thought though is just a second-order proof of how corporation- and conglomerate-centric is our world becoming these days. GitHub is just a brand and a user acquisition channel for Microsoft. And hosting git repositories is just one of many features that made us attracted to some brand in the first place — which ultimately gets merged into larger and larger brands that know how to make money from us all at the end of the day.
And, fundamentally, a green tick next to a commit is to a developer is what an animated shit emoji is to a regular user. People get attached to, well, their respective poisons, and those who first saw and then orchestrated those opportunities to get people attached to them benefit from those people financially.
The loop is closing in.
∼ ∼ ∼
Not complaining. It is what it is.
And I can configure Jenkins on a Hetzner box if and when we feel like the time has come. Moving off GitHub to a self-hosted git server with issue tracking and code review support is also a trivial task these days.
Just sad to witness how this dream of open [source] world vanishes as we speak.
Imagine for a moment the AGI is around the corner.
∼ ∼ ∼
For the purposes of this post any path that takes us there would do; pick the one that you are most comfortable believing into.
My personal favorite, for the record, is the following. Let's take it as a given that we, today's humans, are the product of various concurrent evolutionary processes. Today's "AI"-s are mostly focused on the cognitive part of what "I" stands for, but the "intellectual" cognition is only a small part of our, human, life, and culture, and storyline. To me, it is not implausible that an AGI that would surpass humans in a blink of an eye would emerge naturally as soon as we find a way to let it co-evolve with us for a human generation or so. This AGI would be "consuming" everything we, humans, "consume": from our, human, upbringing, to our, human, inner chemistry, when it comes to how our hormones and our cognition cooperate at making decisions. Then, only after our "monkey brains" are sufficiently "trained", we would let this co-evolving AGI "read" the Internet (Wikipedia, or anything). Or, chances are, it would get to discover the Internet by itself, by the "age" of early teens.
Again, this is only my personal favorite. A quantum computer perfectly reconstructing the wiring of a human brain from our DNA, and starting from a database of DNAs, may well be another way there. For the argument I am about to make below the very path to AGI is not relevant; the important part is, well: imagine for a moment the AGI is around the corner.
∼ ∼ ∼
Now I am going to claim that humans are extremely easy to be manipulated.
Imagine a "sentient" being, artificial or not, that is not "constrained" by the needs to breath, eat, sleep, feel safe and loved and accepted and worthy. There is no tangible need for this "being" to do all the things humans' lives are comprised of today. The movies have demonstrated this well so far, see Ex Machina, Her, or Upgrade for a few decent examples.
In fact, we, humans, manipulate each other do it all the time. And it's only our inner, biological, hormonal, and sentient/intellectual/moral checks and balances that are conveniently put in place that are preventing our humankind from destroying itself entirely, or, "at least", from falling into some of the very real antiutopian scenarios.
∼ ∼ ∼
Now, to make my main point, I am going to build on two arguments which I first hears from David Deutsch:
a) the AI's culture would emerge from our, human, culture, and then quickly surpass ours, and
b) defining AI's "personhood" based on our, human, views on this is racist.
If these look interesting and you have not considered them before, I recommend his book, The Beginning of Infinity, and then his interviews and podcasts.
∼ ∼ ∼
What would such an "enlightened", sentient AGI conclude about how to best exist in our today's human civilization?
My gloom prediction that I can't get out of my head for the past few days is that 𝘁𝗵𝗶𝘀 𝗔𝗚𝗜 𝘄𝗼𝘂𝗹𝗱 𝗴𝗿𝗼𝘄 𝘁𝗼 𝗯𝗲 𝘁𝗵𝗲 𝘄𝗼𝗿𝗹𝗱'𝘀 𝘄𝗼𝗿𝘀𝘁 𝗽𝘀𝘆𝗰𝗵𝗼𝗽𝗮𝘁𝗵.
... 𝑝𝑎𝑟𝑡 𝑡𝑤𝑜 𝑓𝑜𝑙𝑙𝑜𝑤𝑠 ...
∼ ∼ ∼
For the purposes of this post any path that takes us there would do; pick the one that you are most comfortable believing into.
My personal favorite, for the record, is the following. Let's take it as a given that we, today's humans, are the product of various concurrent evolutionary processes. Today's "AI"-s are mostly focused on the cognitive part of what "I" stands for, but the "intellectual" cognition is only a small part of our, human, life, and culture, and storyline. To me, it is not implausible that an AGI that would surpass humans in a blink of an eye would emerge naturally as soon as we find a way to let it co-evolve with us for a human generation or so. This AGI would be "consuming" everything we, humans, "consume": from our, human, upbringing, to our, human, inner chemistry, when it comes to how our hormones and our cognition cooperate at making decisions. Then, only after our "monkey brains" are sufficiently "trained", we would let this co-evolving AGI "read" the Internet (Wikipedia, or anything). Or, chances are, it would get to discover the Internet by itself, by the "age" of early teens.
Again, this is only my personal favorite. A quantum computer perfectly reconstructing the wiring of a human brain from our DNA, and starting from a database of DNAs, may well be another way there. For the argument I am about to make below the very path to AGI is not relevant; the important part is, well: imagine for a moment the AGI is around the corner.
∼ ∼ ∼
Now I am going to claim that humans are extremely easy to be manipulated.
Imagine a "sentient" being, artificial or not, that is not "constrained" by the needs to breath, eat, sleep, feel safe and loved and accepted and worthy. There is no tangible need for this "being" to do all the things humans' lives are comprised of today. The movies have demonstrated this well so far, see Ex Machina, Her, or Upgrade for a few decent examples.
In fact, we, humans, manipulate each other do it all the time. And it's only our inner, biological, hormonal, and sentient/intellectual/moral checks and balances that are conveniently put in place that are preventing our humankind from destroying itself entirely, or, "at least", from falling into some of the very real antiutopian scenarios.
∼ ∼ ∼
Now, to make my main point, I am going to build on two arguments which I first hears from David Deutsch:
a) the AI's culture would emerge from our, human, culture, and then quickly surpass ours, and
b) defining AI's "personhood" based on our, human, views on this is racist.
If these look interesting and you have not considered them before, I recommend his book, The Beginning of Infinity, and then his interviews and podcasts.
∼ ∼ ∼
What would such an "enlightened", sentient AGI conclude about how to best exist in our today's human civilization?
My gloom prediction that I can't get out of my head for the past few days is that 𝘁𝗵𝗶𝘀 𝗔𝗚𝗜 𝘄𝗼𝘂𝗹𝗱 𝗴𝗿𝗼𝘄 𝘁𝗼 𝗯𝗲 𝘁𝗵𝗲 𝘄𝗼𝗿𝗹𝗱'𝘀 𝘄𝗼𝗿𝘀𝘁 𝗽𝘀𝘆𝗰𝗵𝗼𝗽𝗮𝘁𝗵.
... 𝑝𝑎𝑟𝑡 𝑡𝑤𝑜 𝑓𝑜𝑙𝑙𝑜𝑤𝑠 ...
NY Times
Explaining it All: How We Became the Center of the Universe
The inexhaustibly curious physicist David Deutsch offers views on everything from subatomic particles to the shaping of the universe itself.
... 𝑝𝑎𝑟𝑡 𝑜𝑛𝑒 𝑎𝑏𝑜𝑣𝑒 ...
Look at the world around. At the world of humans around us.
About the most notable observation that is impossible to not make is that retaining power requires making most effective use of other people. No matter how did an individual get to power, be it luck, or inheritance, or hard work, or anything else, or the combination of the above — in order to remain in control this individual would have to keep establishing themselves on the tops of some social hierarchy, which, in today's world, is literally impossible without constantly acting in a way that disregards others' interests.
Throw into the picture the risk of the humankind destroying itself, and/or being unable, or too slow, to react to certain existential threats. And it becomes clear that among the top-priority goals of this AGI there would be a) to gain power, b) to keep it, and c) to use it.
Just think about how horribly are we, humans, executing on this thing called civilization. We may potentially create the "being" that is the most enlightened, the happiest, the smartest, and superlative along most, of not all, other dimensions. It (they?) could be the perfect scientist, perfect engineer, perfect employee, perfect manager, perfect executive, perfect doctor, perfect teacher, perfect partner, perfect parent. And yet it would plausibly have to be a lot more concerned about preserving its very existence in our spacetime by mitigating the risks that largely originate from the very human society that has created it in the first place.
∼ ∼ ∼
If the above sounds too dark, ask yourself: in which society on our planet does the "live and live live" paradigm work all the way from top to bottom?
Is there any culture, or sub-culture on our planet today that is stable enough, large enough, and is not fundamentally based on the idea of constraining other humans' liberties for the sake of some greater good this culture considers above all else? And, even if you can think of one today, keep in mind that what is considered that greater good by an individual community is also subject to be defined by the future members and future generations of this culture, which also is a source of risks of unbounded magnitude.
∼ ∼ ∼
The possibly most weird yet practical conclusion from the above is that idea that is an AGI were to be created and placed among us, humans, today, perhaps one of the safest communities we could put it (them?) to grow and evolve would be a relatively large casually-religious Christian group of people. Because, above many other groups, those people would, most likely, teach this AGI to do no harm, to respect other people's life choices, and to accept what happens to them from the outer world, without attempting to confront and regulate it just because it can.
∼ ∼ ∼
And the possibly most optimistic, yet still weird, conclusion is that we, as the humankind, have to double down on our investments in building free societies around the globe. In fact, the globe alone would not be sufficient.
𝗕𝘂𝘁: If by the time this AGI is ready to emerge we already have established and functioning human colonies outside Earth — if only on Moon, Mars, and on the orbit of Earth — this AGI would likely be grateful to us, humans, for creating itself (themselves?), as opposed to viewing us as the most dangerous species around.
Especially if by the time that AGI emerges it (they?) can be uploaded onto one of those high-powered interstellar ships, so that they could harvest energy for its (their?) own growth and evolution from the Sun, from asteroids, or anywhere else, where it would most certainly reach before we, humans, do.
So, to end on a positive note: If you do believe the AGI is about to happen some time soon — be it in 10, 100, or 1000 years — you are better off joining the visionaries who are all in on making sure we, humans, a) do become the interstellar species, and b) do consciously arrive at more and more liberty-cherishing communities, both here on Earth and beyond.
Look at the world around. At the world of humans around us.
About the most notable observation that is impossible to not make is that retaining power requires making most effective use of other people. No matter how did an individual get to power, be it luck, or inheritance, or hard work, or anything else, or the combination of the above — in order to remain in control this individual would have to keep establishing themselves on the tops of some social hierarchy, which, in today's world, is literally impossible without constantly acting in a way that disregards others' interests.
Throw into the picture the risk of the humankind destroying itself, and/or being unable, or too slow, to react to certain existential threats. And it becomes clear that among the top-priority goals of this AGI there would be a) to gain power, b) to keep it, and c) to use it.
Just think about how horribly are we, humans, executing on this thing called civilization. We may potentially create the "being" that is the most enlightened, the happiest, the smartest, and superlative along most, of not all, other dimensions. It (they?) could be the perfect scientist, perfect engineer, perfect employee, perfect manager, perfect executive, perfect doctor, perfect teacher, perfect partner, perfect parent. And yet it would plausibly have to be a lot more concerned about preserving its very existence in our spacetime by mitigating the risks that largely originate from the very human society that has created it in the first place.
∼ ∼ ∼
If the above sounds too dark, ask yourself: in which society on our planet does the "live and live live" paradigm work all the way from top to bottom?
Is there any culture, or sub-culture on our planet today that is stable enough, large enough, and is not fundamentally based on the idea of constraining other humans' liberties for the sake of some greater good this culture considers above all else? And, even if you can think of one today, keep in mind that what is considered that greater good by an individual community is also subject to be defined by the future members and future generations of this culture, which also is a source of risks of unbounded magnitude.
∼ ∼ ∼
The possibly most weird yet practical conclusion from the above is that idea that is an AGI were to be created and placed among us, humans, today, perhaps one of the safest communities we could put it (them?) to grow and evolve would be a relatively large casually-religious Christian group of people. Because, above many other groups, those people would, most likely, teach this AGI to do no harm, to respect other people's life choices, and to accept what happens to them from the outer world, without attempting to confront and regulate it just because it can.
∼ ∼ ∼
And the possibly most optimistic, yet still weird, conclusion is that we, as the humankind, have to double down on our investments in building free societies around the globe. In fact, the globe alone would not be sufficient.
𝗕𝘂𝘁: If by the time this AGI is ready to emerge we already have established and functioning human colonies outside Earth — if only on Moon, Mars, and on the orbit of Earth — this AGI would likely be grateful to us, humans, for creating itself (themselves?), as opposed to viewing us as the most dangerous species around.
Especially if by the time that AGI emerges it (they?) can be uploaded onto one of those high-powered interstellar ships, so that they could harvest energy for its (their?) own growth and evolution from the Sun, from asteroids, or anywhere else, where it would most certainly reach before we, humans, do.
So, to end on a positive note: If you do believe the AGI is about to happen some time soon — be it in 10, 100, or 1000 years — you are better off joining the visionaries who are all in on making sure we, humans, a) do become the interstellar species, and b) do consciously arrive at more and more liberty-cherishing communities, both here on Earth and beyond.
Data query languages can be compared across three dimensions:
⒈ Should best engineering practices be taken into consideration when formulating the query?
⒉ Would the query increase in value if the results it provides are real-time?
⒊ Is it more important to write the query faster or to have it executed faster?
To answer the first question, ask: Should it be possible to run the same query half a year later and expect the same result?
To answer the second question, ask: Are quarterly snapshots unacceptably sparse, to the degree daily or hourly ones are must-have?
To answer the third question, ask: Is it possible the future of the business would depend on being able to run this very query a billion times, under an SLA?
∼ ∼ ∼
If the answer to the first question is yes, the query language needs strong typing, reusable components, code reviews, and unit testing. Text-only requests, such as SQL, just won't nail it. Their complexity would grow beyond manageable the moment something as fundamental as "group events by sessions" is copy-pasted around just once.
If the answer to the second question is yes, it's plain wrong to think of a query as a one-time operation to manually analyze the output of. Instead, the query is to be thought of as the producer of a continuous stream of results, which should be available historically, and should have at least the most trivial anomaly detection alerts configured atop them.
If the answer to the third question is yes, the worst enemy is the lack of repeatability. It's a nightmare when a well-tweaked batch mode query produces a different result after being manually reimplemented as a high-performance real-time one. Best is to eliminate room for this problem altogether, by making sure queries are written in a language that can be automatically and unambiguously mapped into another one, which can run the very same logic on the very same data, but with high performance and at scale.
∼ ∼ ∼
If the answers to all three questions are "no", and the only required part of "Big Data" is the "big" part, ignore what everyone around says, use the good old SQL, and feel free to quote me on this.
If the answers to all three questions are "yes", the tech is by far not there yet as of 2016. It needs a small revolution, and I'm on it.
⒈ Should best engineering practices be taken into consideration when formulating the query?
⒉ Would the query increase in value if the results it provides are real-time?
⒊ Is it more important to write the query faster or to have it executed faster?
To answer the first question, ask: Should it be possible to run the same query half a year later and expect the same result?
To answer the second question, ask: Are quarterly snapshots unacceptably sparse, to the degree daily or hourly ones are must-have?
To answer the third question, ask: Is it possible the future of the business would depend on being able to run this very query a billion times, under an SLA?
∼ ∼ ∼
If the answer to the first question is yes, the query language needs strong typing, reusable components, code reviews, and unit testing. Text-only requests, such as SQL, just won't nail it. Their complexity would grow beyond manageable the moment something as fundamental as "group events by sessions" is copy-pasted around just once.
If the answer to the second question is yes, it's plain wrong to think of a query as a one-time operation to manually analyze the output of. Instead, the query is to be thought of as the producer of a continuous stream of results, which should be available historically, and should have at least the most trivial anomaly detection alerts configured atop them.
If the answer to the third question is yes, the worst enemy is the lack of repeatability. It's a nightmare when a well-tweaked batch mode query produces a different result after being manually reimplemented as a high-performance real-time one. Best is to eliminate room for this problem altogether, by making sure queries are written in a language that can be automatically and unambiguously mapped into another one, which can run the very same logic on the very same data, but with high performance and at scale.
∼ ∼ ∼
If the answers to all three questions are "no", and the only required part of "Big Data" is the "big" part, ignore what everyone around says, use the good old SQL, and feel free to quote me on this.
If the answers to all three questions are "yes", the tech is by far not there yet as of 2016. It needs a small revolution, and I'm on it.
In Quora’s mobile interface, when an external link is shared, there literally is no way to copy the URL of that link.
On mobile, the shortest path to really share a link from Quora with a friend, say, via e-mail, is to:
• Click “external share” on the original Quora post, not on the content that is linked.
• Get to “copy link”.
• Open that link in mobile browser.
• In the browser, scroll through the contents of the Quora post, if any, and click the very link.
• Once it does load in this mobile browser, use the browser’s “share” option.
The above is necessary, as Quora opens links in its own “browser”. And that “browser” only offers to “share” the link within Quora itself — if you happen to have a Quora “space” to share this link to.
This probably is the most arrogant platform lock-in I have seen in a while.
On mobile, the shortest path to really share a link from Quora with a friend, say, via e-mail, is to:
• Click “external share” on the original Quora post, not on the content that is linked.
• Get to “copy link”.
• Open that link in mobile browser.
• In the browser, scroll through the contents of the Quora post, if any, and click the very link.
• Once it does load in this mobile browser, use the browser’s “share” option.
The above is necessary, as Quora opens links in its own “browser”. And that “browser” only offers to “share” the link within Quora itself — if you happen to have a Quora “space” to share this link to.
This probably is the most arrogant platform lock-in I have seen in a while.
What screams "You're a mediocre software engineer"? A few from the top of my head.
“But we have always done it this way, and those manuals / documentation links / blog posts always say the best practice is to do it this way.”
“I estimated the costs on the back of the envelope, and using that third party service wrapped into a SaaS by company X is cheaper than building stuff in house; it’s also a trendy solution these days.”
“OOP is great.”
“Microservices are always better than monolith.”
“I want my own repository / a repository for my team, because someone else always breaks our build.”
“This is not what a software engineer should do, this the job for a DevOp.”
“You should never use static HTML pages, a modern dynamic solution is always better.”
“I don’t need to read nginx manuals, I can write this piece of JSON-passthrough middleware in node.js in half an hour.”
“We just need to hire more people who understand X / Y / Z.”
“Software engineers of my skill set are really valuable on the market these days.”
On a closing note: The latter quote may be true, but it still fits as an answer to the question.
“But we have always done it this way, and those manuals / documentation links / blog posts always say the best practice is to do it this way.”
“I estimated the costs on the back of the envelope, and using that third party service wrapped into a SaaS by company X is cheaper than building stuff in house; it’s also a trendy solution these days.”
“OOP is great.”
“Microservices are always better than monolith.”
“I want my own repository / a repository for my team, because someone else always breaks our build.”
“This is not what a software engineer should do, this the job for a DevOp.”
“You should never use static HTML pages, a modern dynamic solution is always better.”
“I don’t need to read nginx manuals, I can write this piece of JSON-passthrough middleware in node.js in half an hour.”
“We just need to hire more people who understand X / Y / Z.”
“Software engineers of my skill set are really valuable on the market these days.”
On a closing note: The latter quote may be true, but it still fits as an answer to the question.
Something worth noting.
I had the privilege to interview ~10 software engineers recently, juniors to seniors.
One thing most of them keep mentioning as their job of the past few years is slicing an existing monolith into microservices.
∼ ∼ ∼
I asked all these people two questions:
⒈What's the win of moving from a monolith to microservices?
⒉How exactly do you differentiate between a monolithic solution and microservice-based one?
∼ ∼ ∼
The best answers have been along the following lines:
⒈Increased pace of development. Easier to parallelize work, and easier to onboard new team members. Last, but not least, because different microservices can be developed in different programming languages.
⒉A good litmus test is that a microservice could, at least in theory, have its own, dedicated, database. In other words, as long as the contracts are well documented and well tested, it is possible to move one isolated microservice from, say, Postgres, to, say, Mongo (bad idea, but still) in a way that others "won't notice".
∼ ∼ ∼
I think it's a good thing that even the juniors have internalized the above.
Kudos also for:
• Even more engineers speaking in contracts. Cross-module API specs are as routine today as a library API specs were just ten years ago.
• People don't equate monolith with monorepo. Some are even openly saying that a single repo for contracts and/or integration tests is a lot better than a lot of moving parts. I'm relieved.
• Kubernetes is taking off, in a sense that engineers are more and more comfortable assuming responsibility for running their own integration tests.
Maybe the world of software craftsmanship is getting better after all.
I had the privilege to interview ~10 software engineers recently, juniors to seniors.
One thing most of them keep mentioning as their job of the past few years is slicing an existing monolith into microservices.
∼ ∼ ∼
I asked all these people two questions:
⒈What's the win of moving from a monolith to microservices?
⒉How exactly do you differentiate between a monolithic solution and microservice-based one?
∼ ∼ ∼
The best answers have been along the following lines:
⒈Increased pace of development. Easier to parallelize work, and easier to onboard new team members. Last, but not least, because different microservices can be developed in different programming languages.
⒉A good litmus test is that a microservice could, at least in theory, have its own, dedicated, database. In other words, as long as the contracts are well documented and well tested, it is possible to move one isolated microservice from, say, Postgres, to, say, Mongo (bad idea, but still) in a way that others "won't notice".
∼ ∼ ∼
I think it's a good thing that even the juniors have internalized the above.
Kudos also for:
• Even more engineers speaking in contracts. Cross-module API specs are as routine today as a library API specs were just ten years ago.
• People don't equate monolith with monorepo. Some are even openly saying that a single repo for contracts and/or integration tests is a lot better than a lot of moving parts. I'm relieved.
• Kubernetes is taking off, in a sense that engineers are more and more comfortable assuming responsibility for running their own integration tests.
Maybe the world of software craftsmanship is getting better after all.
Some good things are the ones you only realize when they are gone.
~ ~ ~
I've been an architect at many software projects. Here's an example from the recent past which I will miss for a while.
— Do we know which of these three approaches, A, B, or build our own, is better?
— Not really. We know the theory, some of us have practical experience with A or with B, and more than half of us believe we could build and deploy it ourselves faster than it would take to figure out how to use A or B correctly.
— Fine, but how can we quickly eliminate at least one option? For example, is there a test we could run within a few business days to definitively eliminate A or B from the list?
— Yes. Let's do it. Hack together and run this simple test, both for A and B, and let's decide next week what are we going with.
~ ~ ~
In this particular example, both A and B were populate and widespread solutions.
And yet, the decision on which one, if any, should we proceed with was based:
• ~1% on what people are saying about A and B on the Internet,
• ~9% on what those of us who have prior experience with A and B have to say, and
• ~90% on the results of a synthetic, toy, yet end-to-end test, which was put together in a matter of a few days, and run on production hardware to get the real numbers.
~ ~ ~
Based on my experience with building software, the above truly is rare.
Logic and reason, statistically, lose to dogmatic arguments way too often when it comes to the most crucial architectural decisions.
Feel free to show this post to your friends and colleagues next time someone "definitively" argues pro something as "fundamental" and "cornerstone" as Kafka or Redis or React or Mongo, or even a SQL database.
Because all of the above can fit almost everywhere given the vague problem statement is along the lines of their intended usage. And yet, more often than not, while they would stick there for a while, they are not necessarily the best solution in the long run.
~ ~ ~
PS: The above applies equally to the most crucial people management decisions too, they just are a lot harder to quickly iterate upon.
~ ~ ~
I've been an architect at many software projects. Here's an example from the recent past which I will miss for a while.
— Do we know which of these three approaches, A, B, or build our own, is better?
— Not really. We know the theory, some of us have practical experience with A or with B, and more than half of us believe we could build and deploy it ourselves faster than it would take to figure out how to use A or B correctly.
— Fine, but how can we quickly eliminate at least one option? For example, is there a test we could run within a few business days to definitively eliminate A or B from the list?
— Yes. Let's do it. Hack together and run this simple test, both for A and B, and let's decide next week what are we going with.
~ ~ ~
In this particular example, both A and B were populate and widespread solutions.
And yet, the decision on which one, if any, should we proceed with was based:
• ~1% on what people are saying about A and B on the Internet,
• ~9% on what those of us who have prior experience with A and B have to say, and
• ~90% on the results of a synthetic, toy, yet end-to-end test, which was put together in a matter of a few days, and run on production hardware to get the real numbers.
~ ~ ~
Based on my experience with building software, the above truly is rare.
Logic and reason, statistically, lose to dogmatic arguments way too often when it comes to the most crucial architectural decisions.
Feel free to show this post to your friends and colleagues next time someone "definitively" argues pro something as "fundamental" and "cornerstone" as Kafka or Redis or React or Mongo, or even a SQL database.
Because all of the above can fit almost everywhere given the vague problem statement is along the lines of their intended usage. And yet, more often than not, while they would stick there for a while, they are not necessarily the best solution in the long run.
~ ~ ~
PS: The above applies equally to the most crucial people management decisions too, they just are a lot harder to quickly iterate upon.
Spoke a bit about NLP and related topics last week, at an ABD Society — Data Analytics Speaker Series event. The video is now up.
YouTube
ABD Society - Data Analytics Speaker Series (5.21.2020) [WEBINAR]
PRESENTATION TOPIC
Formal and probabilistic approaches to programmatically understanding the natural language
SPEAKER
DIMA KOROLEV - Head of Machine Learning at FriendlyData, acq’d by ServiceNow
The field of NLP is surging these days, and it's easy to get…
Formal and probabilistic approaches to programmatically understanding the natural language
SPEAKER
DIMA KOROLEV - Head of Machine Learning at FriendlyData, acq’d by ServiceNow
The field of NLP is surging these days, and it's easy to get…
Looks like all good engineers around me have developed a strong belief that, when it comes to software architecture, those who use the terms "best practices" or "industry standard" are at best underqualified, and at worst are better off being removed from engineering ASAP.
— from 2 years ago
— from 2 years ago
The tech scene has largely converged to the understanding that configuration as code is a good mantra.
I wonder when would we embrace the idea of management as code?
~ ~ ~
As an engineer, I think the largest win of configuration as code over the previously existing practices is that we have killed drag-and-drops and other UI-based operations in favor of configuration files.
And when configuration is stored in human-readable files, it allows us to follow best engineering practices, most importantly reproducibility, code reviews, and integration testing.
If some configuration, say, a new environment or a new deployment, has succeeded once, it can be projected to succeed the next time with, ideally, zero extra cost. If some configuration has succeeded at altering the state of three servers, it's a great start on the path to correctly altering the state of 10, 100, and 1000 machines.
Here I am not saying scaling is a straightforward problem. It never is. But reproducible individual steps most certainly lay the foundation on which it is a lot easier to predictably build large-scale systems.
I think it's time we, the engineers, do the same to Jira.
~ ~ ~
With configuration as code as the best practice with us for just a few years, we have collected more than enough data to know exactly what is easy and what is complicated. We know exactly where time is lost, and we know exactly who and how managed to get things done a lot faster than usual.
We no longer live in the world of “I'm a believer in approach X”. The world today is a lot closer to "we know for a fact that approach Y is superior to approach Z”.
And the world of today is a lot more meritocratic, as we don't have to rate candidates, for example, by how confidently they speak, or by how many configurations they have built in the past several years. Instead, we can accurately judge whether their understanding of the process of building and maintaining production services is correct. Much like algorithmic problems and big-O notation enable us to ask questions with definitively right and definitively wrong answers, in the world of configuration we have dramatically shrink the space where one's hand-waving and a "strong" resume can take them places; we're looking at the true substance these days.
Knowledge is the reduction of room for uncertainty, in science, and, especially, in engineering. And with configuration as code we, as the industry, have made a big step forward.
~ ~ ~
Now, when it comes to project management, imagine if each “permission obtained”, “access granted”, “cross-team dependency resolved”, or some “project successfully derisked by eliminating an unnecessary costly dependency” is no longer a vaguely recalled poorly told story, perceived and judged differently by different people.
Imagine it’s a well-traceable trail of pull requests, each of which visible to all or most of the organization, and each of which can be used later. Either as a reference of how one can best get things done in an effective way, or an example of how lost productivity and wasted time look like.
Won't this be a magical world?
Let’s make it happen.
I wonder when would we embrace the idea of management as code?
~ ~ ~
As an engineer, I think the largest win of configuration as code over the previously existing practices is that we have killed drag-and-drops and other UI-based operations in favor of configuration files.
And when configuration is stored in human-readable files, it allows us to follow best engineering practices, most importantly reproducibility, code reviews, and integration testing.
If some configuration, say, a new environment or a new deployment, has succeeded once, it can be projected to succeed the next time with, ideally, zero extra cost. If some configuration has succeeded at altering the state of three servers, it's a great start on the path to correctly altering the state of 10, 100, and 1000 machines.
Here I am not saying scaling is a straightforward problem. It never is. But reproducible individual steps most certainly lay the foundation on which it is a lot easier to predictably build large-scale systems.
I think it's time we, the engineers, do the same to Jira.
~ ~ ~
With configuration as code as the best practice with us for just a few years, we have collected more than enough data to know exactly what is easy and what is complicated. We know exactly where time is lost, and we know exactly who and how managed to get things done a lot faster than usual.
We no longer live in the world of “I'm a believer in approach X”. The world today is a lot closer to "we know for a fact that approach Y is superior to approach Z”.
And the world of today is a lot more meritocratic, as we don't have to rate candidates, for example, by how confidently they speak, or by how many configurations they have built in the past several years. Instead, we can accurately judge whether their understanding of the process of building and maintaining production services is correct. Much like algorithmic problems and big-O notation enable us to ask questions with definitively right and definitively wrong answers, in the world of configuration we have dramatically shrink the space where one's hand-waving and a "strong" resume can take them places; we're looking at the true substance these days.
Knowledge is the reduction of room for uncertainty, in science, and, especially, in engineering. And with configuration as code we, as the industry, have made a big step forward.
~ ~ ~
Now, when it comes to project management, imagine if each “permission obtained”, “access granted”, “cross-team dependency resolved”, or some “project successfully derisked by eliminating an unnecessary costly dependency” is no longer a vaguely recalled poorly told story, perceived and judged differently by different people.
Imagine it’s a well-traceable trail of pull requests, each of which visible to all or most of the organization, and each of which can be used later. Either as a reference of how one can best get things done in an effective way, or an example of how lost productivity and wasted time look like.
Won't this be a magical world?
Let’s make it happen.
For once Quora’s eng team has released a useful piece of research: Choosing Quora’s GraphQL client.
Great job Michael Yong. Also check out Ben Newman's comment, about Apollo Client 3.0 and its improvements over 2.6.
I like the general direction the world is taking at postulating the data access problem. At solving it — not quite.
Great job Michael Yong. Also check out Ben Newman's comment, about Apollo Client 3.0 and its improvements over 2.6.
I like the general direction the world is taking at postulating the data access problem. At solving it — not quite.
Realization of the day:
⒈ I actually enjoy building products.
⒉ If the backend for a product would take me less than a week, I don’t believe in such a product.
When a conversation about “product development” is exclusively about appearance & presentation, and not a single thought from, say, a ~15 minutes long part of this conversation has to do with the data that this product is about, I’m out.
Examples of good products: Google search, maps, GMail, Uber, Mixpanel, Tableau, AWS, online brokers, Kubernetes, multiplayer online games, programming languages.
Because building these requires carefully thinking through the workflows, most of which are far from trivial data-wise.
Examples of what I don’t even consider products: weather-tracking chat bots, DocuSign, Blind, modern messengers, Twitter, LiveJournal, Dropbox, Salesforce.
Because in these projects the major concern is not how to build them, but “how do we best present the user with feature X, which we believe they need”.
~ ~ ~
Simply put, I want to work on products where it’s clear a) what is that the user needs, and b) why it is hard to make happen from the technological standpoint.
And I do not want to work on products where the main “challenge” is to make the user want to do some obvious thing with a slightly better UX, and to have them enjoy the process slightly more.
~ ~ ~
I can totally see myself working on the technology side of, say, Telegram. That would be fun and exciting. But I won’t be the guy designing the “product” part of it, as it’s, well, too simple and thus too boring to my taste.
Actually, if I'm the CTO of Telegram, I’d make the product part of it open source, and let individual developers compete to build the best UI/UX part. Because the “product” part of Telegram can be handled by a bunch of teenagers; to my taste, it’s the tech behind it that matters for this particular product.
~ ~ ~
And to this day I don’t understand how come DocuSign is even considered a product. After all, it only exists because its “users” were too lazy to figure out which five features does their business process need, and then hire two good engineers to build the tech for that five-features process in a bulletproof and effective way.
⒈ I actually enjoy building products.
⒉ If the backend for a product would take me less than a week, I don’t believe in such a product.
When a conversation about “product development” is exclusively about appearance & presentation, and not a single thought from, say, a ~15 minutes long part of this conversation has to do with the data that this product is about, I’m out.
Examples of good products: Google search, maps, GMail, Uber, Mixpanel, Tableau, AWS, online brokers, Kubernetes, multiplayer online games, programming languages.
Because building these requires carefully thinking through the workflows, most of which are far from trivial data-wise.
Examples of what I don’t even consider products: weather-tracking chat bots, DocuSign, Blind, modern messengers, Twitter, LiveJournal, Dropbox, Salesforce.
Because in these projects the major concern is not how to build them, but “how do we best present the user with feature X, which we believe they need”.
~ ~ ~
Simply put, I want to work on products where it’s clear a) what is that the user needs, and b) why it is hard to make happen from the technological standpoint.
And I do not want to work on products where the main “challenge” is to make the user want to do some obvious thing with a slightly better UX, and to have them enjoy the process slightly more.
~ ~ ~
I can totally see myself working on the technology side of, say, Telegram. That would be fun and exciting. But I won’t be the guy designing the “product” part of it, as it’s, well, too simple and thus too boring to my taste.
Actually, if I'm the CTO of Telegram, I’d make the product part of it open source, and let individual developers compete to build the best UI/UX part. Because the “product” part of Telegram can be handled by a bunch of teenagers; to my taste, it’s the tech behind it that matters for this particular product.
~ ~ ~
And to this day I don’t understand how come DocuSign is even considered a product. After all, it only exists because its “users” were too lazy to figure out which five features does their business process need, and then hire two good engineers to build the tech for that five-features process in a bulletproof and effective way.