Alex Huras
Seeing Patterns Everywhere

18 Jul 2018



"It's super powerful to see these patterns everywhere, and then you realize, "No, no, no, you're seeing these patterns everywhere, seek help." [Laughs] Everything is a scheduling problem. Everyhing is a set cover problem. Like, how do you move this stuff around?"

Listen to this episode on iTunes/Apple Podcasts

Listen to this episode on Android/Google Play

Download .mp3 (91.6M)

Player by Podigee

Show Notes

Topics: Ad Tech, Career Advancement, Cloud Platforms, Code Schools, Computer Science, Distributed Systems, Education, Hadoop, Internships, Interviewing, Open Source, Software Engineering, Systems Engineering, Teamwork, Team Leadership

Companies

Introduction

Alex Huras was drawn to the tech mecca of San Francisco from his native Ontario after getting a taste of American tech jobs (and salaries) through college internships as part of the University of Waterloo co-op program. Joining Twitter in 2014 with hopes of a bright future, he has stayed through four remarkably turbulent years. And all along he has kept observing and learning. How do teams maintain enough continuity to continue to deliver value through turnover? How does a company decide what to build? How does it avoid expert-level dependencies on open source projects supported by a short list of experts? How does a new engineer learn to solve problems in a big company with complex technical challenges that have been evolving for a decade? How does a young engineer avoid getting locked into skills that will inevitably decay in value? Is it possible to articulate what makes effective engineers effective well enough to hire for it? Alex can describe the mathematical foundations and underlying structure across code, cloud systems and data, but he has also learned the hard lesson that sometimes a beautiful set cover problem fails in the real world because of a mundane failed database backup years earlier. Some episodes of “Using Reflection” start out with a premise. Then there are conversations like this, in which we enter in mid-sentence and the ideas go where they will. I enjoyed the process, and I think you will too.

Guest Bio

Alex spends a lot of time thinking about how to solve problems differently; if for no other reason than as a creative outlet for learning. Through a magnificent stroke of luck; he sometimes does this in a professional capacity. As of writing, this means making advertising data more accessible and reliable at Twitter in San Francisco. Originally from Toronto, he holds an awkwardly shaped degree in Systems Design Engineering from the University of Waterloo.

Links


Transcript

Alex Huras
… sure you have 2000 engineers and yeah, anyone can make twitter over the weekend with like freshmen, like software engineers or whatever. It’s like great, you can do that, but there literally is an infinite amount of work to do and there’s like this possibly extremely large opportunity and it comes down to the same thing, right? Like in theory you want to be like at the edge, at the limit being like, oh, like I’ve got to do this thing and I need to do it as efficiently as possible using pre-built tools.

Host
Right

Alex Huras
If anything, we [at Twitter] have the opposite problem, which is like a whole bunch of people independently … like it’s so big that a whole bunch of people independently go and try to make partial solutions. Like when you’re large enough to have – I think as we were talking about this a little bit earlier – about who is in the right position to be investing in building these platforms and abstractions, and distributed queues are a great example, right? Like if you’re, if you’re a small company and spinning up your own distributed queue, like good luck with that. You’re making a mistake. And I’d argue that, I even even at Twitter scale or, or, or like when you have a team of possibly hundreds of people that you’re able to allocate towards that, it’s still a questionable value proposition just because, because you’re suddenly competing with the big boys, right? Like Yahoo Open sources Pulsar, like Linkedin produces Kafka, like there’s SQS, like all this, all these like giant battle-tested, distributed queuing environments and if you’re going and spinning up your own distributed queue, you better have an extremely good reason.

And I think the other, the other hidden cost here though, and we’re seeing this especially as we’re trying to hire people with this expertise in this, is like very, very few people have expertise in designing and, and like even patching Kafka. You look at the contributors to Kafka like, sure. It’s like hundreds of people. It’s like, yeah, there’s hundreds of people out of an industry that is producing like hundreds of thousands if not millions of engineers like this. It’s actually an extremely small group.

Host
And the long tail of those people are not on the core.

Alex Huras
Yeah, and people are like, “Oh, everyone can contribute to open source.” But some random kid out of college is not making significant patches to the Linux kernel.

Host
Right, or Kafka for that matter. Like the complexity and expertise …

Alex Huras
…and even the testing cost too, right? Like even within within Twitter, like if we want to make changes to the physical topology of our operating environment for something like Kafka or internally event bus, like sure, you make the patch, but in order to actually ascertain whether or not it works like that’s going to cost you $100,000 to run that test. Like it’s not, it’s not like anyone could just spin up an infinitely large cluster and like pump petabytes through it.

Host
Because the testing isn’t just correctness of some function, it’s correctness at operational scale, which is a very hard problem to test.

Alex Huras
It’s like, oh, your number of file handles explodes in this case. That’s a problem. You just lit this machine on fire, when you own it!

Host
And you just went in a circle. That’s why we take the battle-tested option. Just use that because like, yeah, LinkedIn already used it.

Alex Huras
Sure. And I think uh, a bigger, a bigger trend. I mean, I don’t know if this is a trend in industry are just like wisdom, but like if you think you have those problems, think again. Like maybe you do, maybe you’re like a unique snowflake and you have these special problems. But like in general it’s like a modeling problem. Like if you’re, if you’re hitting the limits of Hadoop, you’re probably holding it wrong.

Like, oh, I want to write my own scheduler. It’s like, “Are you sure?” Are you, what if you phrased it as a scheduling problem. Does your operating system solve this for you? Like does Aurora solve this for you? Does Yarns solve this for you?

Host
You can think of partitioning work at all these different levels. And see the commonality of it. Like from one function, one row up to data sets up to multiple jobs …

Alex Huras
And there’s two sides to that coin, right? The one side of, of knowing the fundamentals – and this is also something that is extremely tricky to test in an interviewing environment – and I’d argue also kind of tricky to test even in an actual job environment … because the extent to which you need these sorts of, the ability to multiply recursively and in an enormous number of dimensions analyze these sorts of patterns is possibly … it’s like the path to insanity. Like it’s almost like, oh, it’s super powerful to see these patterns everywhere and then you realize, no, no, no, you’re just you’re seeing these patterns everywhere. Like seek help, right? It’s, everything’s a scheduling problem. Like everything is a set cover problem. Like how do you move this stuff around?

Host
Well the thing is, the reason those fundamentals are fundamental because they are.

Alex Huras
Because it’s math.

Host
You can take a higher level view to solving the problem very quickly and not get thrown off by so many things …

Alex Huras
By minutia.

Host
Like this was so different than the last prolbem and it’s not actually not.

Alex Huras
So I was trying to make my job faster so I checked the scheduler and now I actually I’m yak shaving for like the next …

Host
(Laughs) Right, exactly. But I guess how did we get on this?

Alex Huras
I think the way that this came about was it was like what core assumptions that have governed, how performance of systems work for a long time can be changed by hardware changes and what would happen if disk, if canonically if like disk was faster than RAM, right?

Host
Or as fast. Right?

Alex Huras
Or maybe like comparably fast right. Like now suddenly you don’t have to worry about things being persistent anymore and now all the sudden all of these tools that are built to expect a file system API are suddenly like operating like standalone applications with their own virtualization support.

Host
And you’ve completely broadened the window where you have fast enough access to way way way more memory at a way lower price point than you had before. It’s like a huge architectural change.

Alex Huras
Like even even now, like I mean we can spin up, most people can I think with the power of the cloud. But you can spin up like petabyte memory clusters. It’s like, oh, most of your stuff is like volatile processing and the state that you need to persist, you persist to like some dedicated like distributed system that is like expecting to do that, not your local file system.

Host
Yeah the only time you think about local disk is logs from the local process and even that we could easily change.

Alex Huras
And this is, this is what we were talking about how like structured logging or those sorts of techniques suddenly become really interesting because it’s like, well why aren’t you storing everything in this like mysterious high-performance distributed file system thing, right?

Host
That queryable.

Alex Huras
Yeah. That’s queryable, that has referential integrity.

Host
So is this kind of like everything is Splunk? Is that what this architecture is?

Alex Huras
(Laughs) Maybe. Well this is the mania, right where you see these patterns again. It’s like what if like everything was the JVM or like parallel virtual machine or like some of these esoteric distributed systems.

Host
Cool. Um, you want to actually start?

Alex Huras
Yeah, sure. Let’s talk about life.

Host
Alright. So welcome to “Using Reflection,” a podcast about humans engineering, and I’m here today with Alex Huras. Why don’t you introduce yourself and then we’ll get into the conversation.

Alex Huras
Cool. I’m Alex. I’m a Senior Engineer at Twitter. I work on currently data processing systems predominantly within the revenue or ads ecosystem at Twitter. So time series analytics, a little bit of forecasting, some search index stuff, mostly spread across that domain.

Host
Cool. And a full disclosure, I can say that we worked together a while back when I was at Magnetic and you were still in school at Waterloo and that’s definitely something I wanted to talk about is the program there, but, we can kind of start back to your younger years and you could maybe talk about how you got interested in engineering. Do you remember any formative experiences that drew you to it?

Alex Huras
Sure. So I guess the long story is I used to be really into, I mean it was always kind of interested in engineering or scientific or those sorts of worldviews. My father was an engineer. Yeah, like how things worked. I like building things. I like taking things apart. I think the second is probably the bigger, the bigger engineering story. I think that there’s a Dilbert comic, right? And so that kind of naturally led to like I was interested in math and science. I think it was a fairly traditional cycle there. I think the one possible caveat there was I was really into music. It was the situation where I had to decide between going to school or college for music or systems engineering, and my father is a systems engineer and literally went to the same program and my parents were very supportive, but it’s actually, they were like, so do you want to get a job? Like at some point in your life? And, and the answer there is eventually, like I went and did engineering at Waterloo.

Host
Uh huh. Do you regret that at all?

Alex Huras
No, no, not at all. I think it was, I think it was a great decision. I don’t necessarily think I would have been screwed the other side, but I think especially going back and visiting people that I, that I knew from back then that went into different industries, I think engineering is an excellent choice. The job market is extremely difficult for essentially everyone else right now, and so I think that that had a lot of formative experiences, especially as an introduction into software that I actually did [help me] kind of later. So I was actually a plumber for one term. That was one of the more interesting experiences I’ve had in my life and I don’t regret doing it, but now it happened. I did some finance stuff and eventually kind of taught myself to code during that because it was so mind-numbing.

Host
So, in college?

Alex Huras
Yea. This is still a co op stuff.

Host
So what were you studying? Computer Science?

Alex Huras
So yeah, so I was studying something called system design engineering at Waterloo, which I like to describe as the most vague, undifferentiated form of engineering that Waterloo offers.

Host
So you got to decide what it was.

Alex Huras
Pretty much. I mean what it ended up being for me was five years of Linear Algebra, but it can take different forms for different people in the program.

Host
OK, so how does Linear Algebra map to Systems Engineering?

Alex Huras
Oh how does it not. Everything’s a schedule problem. Everything’s… Everything can be expressed as a matrix transform.

Host
That’s actually kind of true, especially if you’ve ever programmed in APL or something.

Alex Huras
Exactly, right. It’s all numbers all the way down, and so I think the exposure to software ended up just being kind of like it was interesting. And then what ended up kicking it off was I worked, I got an internship I was super excited about at a small tech startup in the barrier called Skygrid at the time and that essentially started me off on a trajectory of where the next stop was Magnetic working with you guys.

Host
Uh huh, well, so you did pretty well on the phone screen you considering you just kind of taught yourself to program.

Alex Huras
Yeah, yeah, I mean I was, I was like writing some what I now realize that it’s absolutely horrifying javascript at Skygrid.

Host
I still remember that phone screen because I asked you the question and you said something like, “Ah, OKw, I see why you’re asking me this.” And then you proceeded to answer it.

Alex Huras
Right. Like use a heap or something, it was something like use a heap.

Host
Yeah like all 30-minute test questions.

Alex Huras
It’s like, would you like me to implement it? Let me jump through these hoops for you.

Host
I tell people that I’m like, just start with which data structure that’s like, yeah, most of them. And does sorting help me?

Alex Huras
Yeah, do I need ordering? Like maybe I do. Yeah. And actually just to slightly tangent there, I think one of the harder things and why interviewing for more senior engineering candidates, like especially like once you can assume that people know the data structures are the next thing is like instead of it being like what data structure to use, it’s like what are the constraints like period, like and that, that question is, it’s sort of like you build your career and trying to solve that question, right?

Host
That’s requirements. That’s the business. That’s the operating environment, that’s the world, that’s the competition.

Alex Huras
Exactly.

Host
Now you’re in the domain of the whole world where I think that’s a big separator from a junior versus more senior. That’s interesting. Okay. So you got the bug. Yeah.

Alex Huras
Yeah.

Host
And then so what, what was your career trajectory like at coming out of school? You decided you wanted to stay in software?

Alex Huras
Yeah, and I don’t necessarily view it as like a, I mean within school there’s always this kind of existential pressure of like not liking software because it’s like … soft. There was, when I was in Waterloo, which may or may not be the situation now, there was definitely a strong bias against incentivizing software engineering or to, to, or like non-software engineering and primary specialists. Like essentially there is a software engineering program at Waterloo. If you’re trying to do those sorts of things in the more conventional engineering stream is frowned upon.

Host
I see. Interesting. Even applying it back to these other disciplines …

Alex Huras
Sure, and there’s a lot of design projects …

Host
But isn’t software having an impact on all these engineering disiplines?

Alex Huras
You’d think so. And I think it just comes down to possibly the, just the personalities of the individual professors that are doing these things, many of which are not in a software environment. And the other part is like software is still kind of this, this like vague, ambiguous. Like it’s not, it’s not. You couldn’t really consider it to be like an applied science, like it’s very much like, and this is something that is maybe a little bit more subtle, but there is software engineering as in solving problems that relate, that you essentially use computers to solve and then there is software engineering as in like writing code, and writing code I don’t believe is like an engineering discipline. It’s an extremely powerful tool in the same way that like speaking English is not like an engineering skill in and of itself. Right. But it’s absolutely necessary for you to be able to operate.

Host
Did you feel that as like a pressure or did you just kind of feel like you were going to do what you wanted to do? Like how susceptible were you to that environment?

Alex Huras
Yeah, I don’t, I don’t necessarily. I mean, I’m sure in retrospect, maybe I was, but like a large part of it was the fact that like, let’s be real. The software engineer, the software engineering industry in the United States pays roughly double if not more than the equivalent thing in Canada and probably similar multiples everywhere else in the world. And so being exposed to that market, especially after I did Magnetic, I worked briefly at Facebook and like, once you’re there you’re there. Like there’s no going back at that point.

Host
So you’re like, I’m in New York making money.

Alex Huras
Yeah. Yeah. And this is the thing too is like you’re paying more than like even as an intern, you’re paid more than what your tuition is like to go to school. And so you do that at facebook, they pay you even more and then you get like a full-time offer and you’re like “Wow, this is ridiculous.” And so the economic incentives were aligned, but I think the second thing is like, I don’t know what it was, but like maybe 50 percent of my graduating class ended up doing software-related things.

Host
Well, when was that.

Alex Huras
This was 2014. Yeah.

Host
Well it’s probably just where a lot of jobs were.

Alex Huras
Yeah. And that’s. And that’s, that’s what it is. It’s like that was the market that was hiring, huge employer

Host
Super aggressively up there I’m sure, because I’ve worked with a lot of people from Waterloo, and I want to get into the co-op thing in a moment but, and just the level of talent and ttitude and also their preparedness because of the co-op program, it’s incredible.

Alex Huras
Yeah, and so you graduate, you’re essentially ready to like you essentially have possibly up to two years of industry experience as a new grad. You’re competing with people who have zero experience, like the stars are aligned.

Host
And all the things you learn working you don’t learn at all in school.

Alex Huras
Sure exactly.

Host
How to work with people, which is something we can get into. How to work in a collaborative code base …

Alex Huras
So like when I wound up starting to work for Twitter in 2014, it ended up being not a huge deal for me I think in terms of like process. If anything I was kind of surprised by like how much less there was.

Host
It was more casual than some places.

Alex Huras
It was more casual than, for example, like what I was doing at Magnetic.

Host
I seem to remember lots of like debating about whether 10:00 AM was too early for the standup.

Alex Huras
Oh yeah those debates still go on I assure you, strangely not when I’m working in New York. So there’s, there’s that thing and I think the, something that takes a lot of, especially in new engineers that come out of CS education or whatever into their first software job by surprise is the fact that there aren’t really problems that you can just like give someone with no expertise and just like have them solve in isolation. Like that’s not the most effective use of your resources. And so if you’re not really, I don’t mean sometimes there are others problems and if so then like those are great, like phenomenal.

Host
So you mean usually the scope is too broad and interconnected …

Alex Huras
the most effective way to solve these problems is not to just like sit down on your own, on your own and just like slam your head into the wall.

Host
Sure. That’s almost certainly true because almost everything about the solution is going to come from a knowledge within the organization.

Alex Huras
Yeah, exactly. Everything’s convoluted, and there’s like … I remember what the first thing that I had to do was figure out some, like the sum of some numbers, and this is like classic accounting problem because the sum of some numbers didn’t add up to the total … like two data sources, combining these things. And it ended up being this like week-long investigation,

Host
Isn’t that every problem in ad tech?

Alex Huras
Yeah, pretty much. It’s, Oh yeah, it’s just a haunting problem really. But essentially at the end of it it was like, “Oh yeah, like the database is out of sync.” Like we lost the master, there’s some failover that was botched like six months ago, and then this data was just lost from six months.

Host
Two hours of data from six months ago.

Alex Huras
Yeah, exactly. And you’re like, “Oh, so what did I learn? I learned that you should just ask this guy.”

Host
And the other thing, the other, the other thing I find in each dream job is you have to learn the code base, right? You have to read a lot of code and understand sort of what’s where and map it to a mental mental model of higher level idea of kind of boxes and arrows of the system and from that to the business and then that makes sense of the names you’re seeing in the code and the functions of the things you’re seeing. And this whole, that’s a whole process that when you’re given assignments which are like write a function or even write a little application, you’re not dealing with that scope, that breadth and needing to kind of have multiple levels of abstraction in your mind.

Alex Huras
Yeah one of the things that blew my mind initially was like this concept of requirements changing over time, right? I mean whoah, whoah right? I mean like you see little pieces of this, but like by over time I mean not like while you’re developing it. I mean like you’re looking at something system and like there are these points in history where like things mean many different things.

Host
Where the columns changed or the inputs we used to this thing changed.

Alex Huras
Yeah. And it’s like, it’s like, why can’t I get the number of impressions on this video across this four-year period? It’s like, oh, because actually they were indexed into a different place because this team was owned but in a different place and there was an acquisition. It’s like, Oh, a serialized, wrong to begin with and it was cheaper to just like change our worldview that it was to like go back and like patch the data. Yeah.

Host
So where, what, what company was this that you started in, in your first job out of school?

Alex Huras
Oh first this was Twitter.

Host
Oh so you have been at Twitter this whole time.

Alex Huras
Yeah.

Host
Huh, interesting.

Alex Huras
It’s been a very interesting ride especially and I mean a lot of people don’t … Like it’s not, it’s not the experience I signed up for it, but I don’t regret it.

Host
Well what did you think you were signing up for?

Alex Huras
Well, I mean there is, I’m not necessarily susceptible to hype, but it was, it was definitely a small-team environment. It was essentially the remnants of Twitter’s BackType acquisition and doing a kind of a full stack ads analytics thing for Twitter. And so like within a couple of weeks there are a lot of changes. Like had IPO’d, they were going to be the next Facebook, that didn’t happen. There was this giant like morale swing …

Host
Like I’m 22 and this is my first job and this is happening …

Alex Huras
Yeah, exactly. And so you’re, you’re kind of like, I learned a lot about how organizations work or don’t work and that I think has been invaluable, but it’s not something that like I would actively look for in a, like, you wouldn’t, you wouldn’t go …

Host
I mean maybe even some chaos.

Alex Huras
Oh yeah, definitely. I mean some people do this and also we’re hiring (laughs), but you wouldn’t dive in and be in this really hard place where there’s all this fresh competition and all this extra hype that’s moving away. There’s like Snapchat and Instagram.

Host
Right. Some people love the idea of a challenge.

Alex Huras
Yeah and so that was not necessarily what I had signed up for. I had signed up for, possibly naively, for like more of the exposure to some interesting technical challenges with an environment of people that were like really neat. And to see that environment like totally shift almost annually over the next three years has been extremely interesting.

Host
But that goes back to what you said about requirements changing or something like a huge difference between working and, well, all education is basically education starts with the assumption that there’s a static set of whatever you’re going to learn and it’s static.

Alex Huras
Yeah, like connect the dots.

Host
This stuff is basically the start of the semester, here’s what we’re going to learn. Whereas working, it’s like – especially I’ve been in startups now really consistently, depending on how you count, seven to 10 years let’s say – things change every day. And it’s what I find really interesting about it is you need this constantly evolving mental model of what the business is, what your job is and how to be …

Alex Huras
Or even what resources are at your disposal. And there’s also, especially even over – and software is super interesting here, but like even over the last three years, like industry, technology changes like the cloud …

Host
Righ, I mean the reason I went back to being hands on was the cloud because I thought to myself, “If I don’t start building stuff this way, I’m going to be totally irrelevant and I have to have the rest of my career.”

Alex Huras
And that’s, and that’s … essentially like I’m, I’m like pretty young, pretty early in my career and that still terrifies me. We interviewed people who have lots of experience, all of it, like all of the marketable experience is irrelevant. And it’s. And it’s like that is …

Host
So let’s stop there for a second.

Alex Huras
I’m afraid.

Host
Don’t you think that those people could learn?

Alex Huras
I definitely, they definitely can. But the question is like how do you set up, how do you set up … like let’s say like you want to hire those people anyway, right? So then the question is how do you set up an environment where they can actually learn that effectively and their value proposition is commensurate with how much you’re going to be paying them? And this, I think this is also kind of like the, the dark side of software engineering in many respects is like if you’re being hired based on skills that you possess as opposed to like how you solve problems or leading teams or whatever, then those skills better be super relevant otherwise it’s overpriced. Right?

Host
Skills is a very hard race to run.

Alex Huras
Right. And so this is, this is the thing where it’s like I had the opportunity even at Magnetic it was like,”Oh do you want to like drop out of school and we’ll pay you a lot of money to like write Map Reduce jobs?” And I was like, “Oh man, this would be really interesting,” except for the fact that like, that would have been an absolutely horrifying career move. Maybe I would have been super excited and like jumped on Spark, jumped on Tapestry, like all these other things, but like, if you don’t, if you just set park there … Like we see people that come out of Oracle or even or like some of these other companies where it’s like you’ve been working here, you’re an industry master at like how to use MySql for example. Like you know everything, you know the ins and outs and it’s like, “Do we run MySql here? No. Like I guess you’re not eligible for this position.” Like unless you’re willing to be like I am, I know I’m willing to take like a 50 percent pay cut and work on something that I’m not good at.

Host
That just came up in the last episode with my friend Corey who’s very much a Postgres specialist and contributor and um, has made a 20-year-plus career now basically specializing as being working with relational databases. But he’s been a consultant almost the whole time. And his point was, if you’re doing what you’re saying …

Alex Huras
You need to be consultant.

Host
Yeah, the consultant fit is “I do specialize, I’m pretty much on the cutting edge, which means I’m better than the people in house.

Alex Huras
Yeah you’re going to pay a premium for this skill set and you’re going to have to jump around …

Host
And when you need me, I’m gonna be totally reliable and better than who you have. And he said the key is if you’re going to focus around skills – the two things maybe are you got to try to always kind of be moving toward where you think opportunities are going to be, and you need to find things where you can be better at than most people.

Alex Huras
Sure. And typically that means super nichey, or almost you can do the opposite, which is like super generalist and maximize transferable skills …

Host
I’m not really interested in you slinging dirt about Twitter per se, but it’s interesting to me that it’s, it’s been a volatile environment, meaning and a sense of there’s been a lot of change going on while you’ve been there. Right. Not, not trying to have a negative connotation. And it’s interesting to me that, you know, your expectations were almost immediately confounded, especially about what it would be like working with teams and all of that. So what have you seen or what do you think you’ve learned about what makes teams effective in, in this kind of situation where there’s a lot of change happening and how you can be as valuable as you can be?

Alex Huras
The biggest problem that I’ve seen or rather like a symptom of teams that are not aligned properly or like not effective I guess in this terminology, is like they don’t necessarily have a common goal or skill set. Like it isn’t … like teams with strong visions and who have people that are essentially dedicated to solving core problems. Like regardless of how they get solved, tend to be extremely effective. They survive through like heavy attrition or like the stock price dropping by 2x or whatever, and it ends up also being easier to hire for in many cases. Right? So like even over like four or five six seven years or whatever, very few teams at most companies in the Bay Area I think had been like totally stable. Right? And yet the teams that I do see around Twitter and around other places that are, have all these things in common. It’s like they’ve got a core mission which is like relatively unambiguous and they use to actually decide whether or not they’re working on [the right] problems. They have access to resources to solve problems with, whether this means using the cloud or like having people, and they’re essentially, they like self-select for people that are kind of aligned. And a lot of it is like, okay, well if you’re a junior engineer, there’s a place for you where you can actually learn from people who are really good at what they’re doing because they’ve been so stably aligned in this direction. If you’re hiring senior people, it’s like, hey look, we’ve got these junior people you want to mentor, like great, here’s some people like make everyone better. And then strategically, even within a big organization, whether it’s Twitter or any other engineering environment, that team becomes almost … You can, you can trust it more. There’s this a team dynamic thing where it’s like you have to build a whole bunch of trust and consistently do that and if people don’t trust each other, then the team collapses and if people don’t … if a team collapses like you can’t deliver. And so there’s this kind of stable repetition of things that people are actually interested in, and that, that typically wins.

The problem then is like, if you’re not in that position, how do you like assemble that from the chaos? Which has happened to various parts within Twitter and other companies as well. If your company is relatively flat and it has this long-standing tradition of like anarchy, then it’s going to be very difficult for you … like those teams might develop organically; they might also disappear organically because they grow together and they’re like, “Wait a minute, I could just like start a consulting shop.” And then the whole team like leaves or company or whatever.

Host
And that goes back to the consistency you were just talking about.

Alex Huras
Yeah, and so you need, you need some direction, whether or not it’s super top-down or whatever like it doesn’t matter. If there’s a direction and people buy into it and also realize that like the company as an abstract entity is actually like incentivizing you to be more successful than it’s like for lack of a better term, synergy or whatever.

Host
Right.

Alex Huras
And I have to be totally honest I have not seen that in its ideal form, but that’s, that’s something that we’d kind of like strive to align people around.

Host
Right. So then what about you personally? How do you identify how you can best fit into that mission once you identify that that mission is there? How do you kind of keep realigning yourself?

Alex Huras
Yeah, and that’s like the big existential question that I don’t necessarily have a good answer for.

Host
Yeah that’s like working right?

Alex Huras
Yeah. So like my, my general answer is I just kind of like work on things that interest me and it just so happens that I’ve been kind of interested in this like data processing domain for enough time that kind of know what I’m doing …

Host
But I mean can you map it back to the goal of your team?

Alex Huras
There’s kind of this weird feedback loop where like I’m interested in solving this problem, like as long as someone, as long as someone says that that problem is worth investing in and suddenly there’s like all these other people that are interested in the same problem, and it’s almost like I was like, we set my roadmap being let’s solve this problem or let’s work with other people to solve this problem. And it’s kind of … and maybe this is why it’s super difficult, but it’s like that there is this virtuous cycle where it isn’t necessarily like I have to consciously align my interests with the business. I think that’s ideal. I think in reality within advertising, you do need that realignment when it comes down to individual teams, like the stability of a common goal being repeated for a long time can exist. Even like Twitter, we’ve got like 2,000 engineers or whatever. It’s like you don’t need 2,000 engineers to have a common direction. You actually only need like 100 of them.

Host
It’s really hard for 2,000 engineers to have a common direction.

Alex Huras
I mean tell me about it (laughs). And that’s something that like Google has to address my problem too, and I think even when you’re, when you’re working through your career, I mean if you’re, if you’re in a position where you have to consider the alignment of 2,000 individuals, like you’re probably not doing software engineering anymore, and for at least where I am in my career, I think it’s enough to have some sort of local stability that you can be like, all right, we’re going to be doing this for two years and this is the, this is the vision for how we’re going to solve these problems. And like you’ve got any resources that you can, that you can find or like that we can provide to solve it. And I like the restriction there being on what we can deliver rather than being like, all right, here’s a bunch of artificial constraints like go solve the problem.

Host
Yeah. And I think, I mean because I’ve also been spending more than the last year plus here on like data and have done that earlier in my career too. And one observation I can make about that is it’s somewhere between infrastructure and product features.

Alex Huras
Oh for sure, yeah.

Host
And so like you can be a little, your goals can be really clear even if all the different features or whatever uses of the data are less clear.

Alex Huras
Sure.

Host
And I think so you can be in this kind of world where you’re just like … your interface to the rest of the businesses is like “I delivered this dataset and it’s consumable by you in these ways.” And so that that’s like a very clear goal.

Alex Huras
And I think the, and this is like the meta-trend for that part of the industry is like there’s a situation, there’s this kind of blurred line between having non-engineers be able to declaratively do these sorts of things and having … and like implementing … and on the engineering side it’s like implementing platforms, that you’d want almost this like global compiler that is like stitching together known data sets. And it’s like, “How do you write that?” It’s like, “Oh I don’t know.”

Host
Yeah right. But that would still be something with a very clear goal that you can define …

Alex Huras
Yes. And so the, I think the danger especially for that – and this is literally like what I’m working on right now – but like the danger for those things is like, again, this kind of blurs the line because why would you write that yourself? Like that’s an expensive project to undertake. Like what are the value adds? You’re trying to essentially justify spending like millions of dollars of, of resources over multiple years trying to solve this problem. Then you have to be like, oh, is that actually a problem? And you’re talking to people who have no idea too, so it’s like, oh, like maybe people don’t, don’t mind writing this themselves. Maybe we were totally happy hiring engineers to do that. Like I haven’t had to think about this before, but maybe it wasn’t as clear as I thought.

Host
Don’t take this the wrong way, but you’re not drawing on 20 years of work experience.

Alex Huras
No, exactly. I have no idea. People ask me these questions like, I don’t know, I just work here. (both laugh)

Host
Right. Interesting. I want to circle back a little on Co-op too. Maybe you could first frame it for people who aren’t familiar with it.

Alex Huras
So the Waterloo Co-op system is essentially, you could think of it as interspersed in your normal undergraduate degree you have compulsory four-month internships that if you don’t complete them to some satisfactory manner, you just don’t graduate. So in engineering at Waterloo this is required, like you can’t get an engineering degree, which is a Bachelor’s of Applied Science, without completing five of six possible co-op internships.

Host
So you went to school for four years.

Alex Huras
Yeah. So the, the undergraduate degree is four years and eight months, so roughly five years.

Host
So you go to school for almost five years and around two of that …

Alex Huras
So the way that my undergrad worked is like you started school in, in the fall of whatever year that was and four months later you like Christmas time or whatever, like during that you have to like find a job and there were resources that help you find a job, but it’s not like one is provided for you. You’re competing with, with all the people in Waterloo and theoretically the world depending on where you’re going and you have whatever experience you have, which after the first four months of university is effectively zero. And this is also why I was a plumber. Yeah, it’s like “Is Facebook going to hire me? No.” And so then you’re four months of working full time. Essentially you’re then rotating every four months until essentially the last year and then you have eight months of solid school.

Host
And each time you have to go and get a job again.

Alex Huras
Yeah.

Host
Does it become more and more competitive? Does it become like a thing?

Alex Huras
So yes and no. I think it’s actually extremely competitive at the beginning because most of the people are not differentiated.

Host
I see.

Alex Huras
At the end of the … Well, especially especially in your equivalent to senior year, you are the most competitive intern on the planet and so like, yeah, sure, you’re competing with other people at Waterloo, but if a tech company wanted to hire interns, they’re just gonna like pick you. I mean generally, I mean that’s kind of a …

Host
And hopefully by then you’ve gotten like a blue chip … like by that time you’ve interned for Facebook or whatever.

Alex Huras
Yeah. And, and that’s the thing is like those are kind of like the jackpot I think in terms of software engineering, but there are a lot of other large-scale employers. So for example, the government. Yeah, government is a big hiring. Like you can do research assistant type things within universities. There’s lots of smaller engineering companes especially for non-software disciplines. But I think the key distinction is like when you graduate, you do graduate later than then you’re essentially people the same age group. Like about by about a year, but at that point you have two years of quote unquote real world experience and if you knew what you were going to do and you aligned yourself so that all those were, for example, like within software that is like two years of industry experience, but being, being like fresh out of school.

Host
And did you find that people, especially the internships you did later, that there was an understanding that you could come back there after you graduated or?

Alex Huras
Uh, yeah, I think every single, every single place I worked was like that.

Host
There you go, so you’re in a good position, you’re not freaking out when you graduate.

Alex Huras
Yeah and I think it’s also like, I’m not sure how much of this is because the tech industry was taking off, but I think in general like, it is like … working at a place is the easiest foot in the door that you could have working there. Right. So like I know a lot of people that were working at companies that were not necessarily converting internships into full-time offers, but you would, having worked there and theoretically performed well then you’re, you’re way better off than like any random person.

Host
Oh, I mean speaking as someone who’s had to hire people, the amount of risk that that covers is like nearly all of it.

Alex Huras
Yeah, part of this is communicated actually to employers in the Co-op system. I’m not sure how much I agree about this, but like there’s like a public rating that you get for every internship.

Host
Ooh!

Alex Huras
Yeah. You may not have seen those when I … but there, I mean some of them are like gamed and you can essentially like elect to not display them and some things. But essentially like the employers that are looking at the Waterloo pool as integrating into the Waterloo system have a lot of power. And so the contract that they kind of implicitly have with the university is like, you have some slots, like you’re going to fill them and you have a lot of information, like more than you would normally have.

Host
Well, the school’s a big part of the school’s value proposition to the students is …

Alex Huras
… is this network right …

Host
… right, of desirable employers. So they are making this two-sided market that is very intersting.

Alex Huras
Yeah that’s ripe to be disrupted. Right. But, and this is, this is kind of the bigger question is like why, and I still get this today, like why does no one else have this?

Host
Right I mean of course that’s where I’m going next. How repeatable and scalable do you think this model is?

Alex Huras
If you, you were to just have the requirement that people go out and find jobs every four months. Like until they graduate that that would, I’m not sure … Like, and this is one thing that you were like I’m not, I wasn’t necessarily in a situation where I was taking advantage of the support network of Waterloo, but that is valuable in and of itself. And so like if you were a school where you want it to have internships, I think the first step is just like don’t do them in the summer. Like have school in the summer. Have your interns compete with effectively n onne in the winter and fall and now suddenly all these companies have like full-time intern support.

Host
I suspect even though you said you weren’t taking advantage of the support, of course you were.

Alex Huras
Sure implicitly there’s like this reputation …

Host
There is a whole infrastructure of processes that are occurring that you had nothing to do with that are still helping you get a job.

Alex Huras
So for example, like my, my earlier stuff that I did … like the first three jobs I effectively found on my own outside of the system. I think Magnetic, I did end up using the system for.

Host
I remember that was set up and I talked to other pople that day.

Alex Huras
And so like that from a recruitment angle makes hiring from Waterloo very efficient. And so like if you were a university and you wanted to implement something like this, like that would probably be a sizable investment to build up that stuff. But I think for a lot of smaller institutions, like if there’s no reason why this cannot work.

Host
You do think it’s … would be beneficial?

Alex Huras
I think so.

Host
It’s like, I’ve been thinking a lot about this. The job market for what we do is so broken.

Alex Huras
It’s nuts.

Host
Along multiple dimensions. There’s not nearly enough people and yet the system that’s currently used to find and select people has huge amount of bias.

Alex Huras
Yup.

Host
It’s leaving out so many sources of people …

Alex Huras
Sure, and I think there is a lot evidence to say that it’s not even that successful.

Host
Right! And almost everyone who goes through it doesn’t think it has much to do with who ends up being good or not. Many hiring managers, you agree with that too. And, and yeah. Great. And I mean I thought that it twice or at least during this conversation when you were talking about speaking to these people with a lot of experience but maybe not the right experience …

Alex Huras
It’s almost like a marketing problem, right?

Host
Well yes. But I also think you need multiple actors in the system providing more training and support

Alex Huras
Sure, of course.

Host
So like your Twitter example of like someone walks in the door who has been shipping code for 25 years but doesn’t have your skills on your random HDFS cloud whatever whatever, like, so take them provisionally for three or six months and train them …

Alex Huras
Yeah and that’s something that’s super interesting because like a lot of the … and so as a Canadian citizen as well, like the immigration system is not set up for provisional employment into the U.S. So if we, if we, if we table the immigrationr-related discussion, which I also have strong opinions about, and a vested interest.

Host
Yeah.

Alex Huras
The like even internally I don’t even know if people like if that … Because there’s nothing stopping companies from taking on provisional employees. Right? I think it’s very tricky for sure. And I think it’s like risk management for employment or whatever and that, maybe, maybe that gives the employer a lot of power. Maybe it doesn’t. Like it’s kind of a …

Host
I don’t know. I think they’re worried about like a training and you leave that kind of thing.

Alex Huras
Sure.

Host
But you know what, like you don’t have that person now.

Alex Huras
Yeah, exactly.

Host
I think it’s up to businesses to retain people by making it a good proposition for people to stay.

Alex Huras
Exactly.

Host
And you start out with some loyalty. If you took someone who wasn’t in a position to work for you and you gave them the opportunity to work for you, you know, you get some loyalty. Also, like, I’ve had, um, benefits that came with requirements. Like when I was at Moody’s and they were paying for my grad school, you know, you had to say for some period of time after that or pay it back to them. So anyway, I just think that that’s an example, and I think this example of the Co-op is another example.

Alex Huras
But I think part of that is also kind of this indentured servitude model which does not like …

Host
But the reason now reason we have that is largely because of how we pay for healthcare.

Alex Huras
Sure.

Host
I mean, speaking of someone who has the only wage earner for a family of four, two family members have a significant health issues and that’s my whole life. I have to be so carefully tied to an employer for the wellbeing of my family. It’s crazy. And so like me being relevant in the job market, it’s not an abstraction at all.

Alex Huras
Well especially if you extend that and you’re like, “Oh, if I’m not tied to an employer, I’m legally not allowed to be in this country anymore.”

Host
(Laughs) Now we’re back to the integration topic.

Alex Huras
No, but it’s, it’s, it’s like, this is why it’s such a huge, I mean all of these things, obviously it’s an extremely complex system. All these things are tied together, but like it’s like even as like an American citizen working at a, like an upstanding business like getting better. It’s like if you’re still thinking about that, can you imagine it’d be like, oh, like I lose my job and like also like I could be like arrested. Right? That’s huge.

Maybe that power distribution was OK, but we need better ways of paying people for work that they do. Part of this. I mean, part of this is kind of the quasi-toxic yet very economically interesting distributed contractor model or somthing, like the Uber’s right?

Host
Are we talking about the gig economy?

Alex Huras
Yeah, the gig economy right? And this is the thing too where it’s like if you were in, and this is the ideal thing for an employer, right? You have some very, very clear definition of what a job is that you want to do. And you’re just like hire people to perform. It’s like a machine. You hire like an Automaton to perform.

Host
Yeah in fact it’s going to be self-driving person before you know it. A self-driving car.

Alex Huras
This is the thing too is like even if you were in a situation where you had to hire software engineers for example, and you’re like, okay, I have some extremely well specified, like you have the test, the golden tests that says like just pass this test. Those jobs are already outsourced, like they don’t exist here. And so when it’s interesting, when you, when you’re talking especially newer engineers and maybe older engineers that are kind of looking for that, like the simplicity of having predefined problems, just like, no, like if you want to do that, we’re going to pay you 10 percent of your salary. Like how do you feel now? Right?

Host
Yeah so the way I look at this is automation is eating the bottom out of the every technology jobs.

Alex Huras
Yeah it’s like any decision problem is on the table.

Host
Like so, right. All of these are the same pressure, right? There’s more and more of a need for software. There’s not enough people, right? So there’s a bunch of startups and initiatives to try to help solve the hiring problem. There’s not nearly enough being done. Education is going to change. Attitude toward hiring people from alternative backgrounds is going to change. Training to support is going to change.

Alex Huras
This is also kind of my, my cynical perspective here, but like there’s all this like learn to code initiatives or whatever and it’s like coding is not the problem. The problem is not that there are not enough people that know how to write a function in javascript or python or whatever your language. It’s like if that was the problem, then we would have compiled away the solution. Right? Like tools will solve that problem. I mean it’s okay to use code as a vehicle to teach problem-solving skills or whatever, but like that should be the super-explicit goal. It’s like actually this is just a trick. It’s like an elaborate illusion that we’re presenting so that you learn math, right? Like this is like this is just a game we’re playing as a society to, to encourage you to rigorously solve these sorts of problems and that is what we’re missing. Or at least that in, in the, in the sorts of problems that I’m solving like that, those are the people that are like, I’m actively looking for and it’s like extremely sparse.

Host
So do you think that’s hard to teach or impossible to teach?

Alex Huras
I don’t think so. I think it’s just like, like video games teach this. Like this is, it’s not a, it’s not like a hard quote unquote hard problem just by the fact that it is like, it is a skill allegedly, but it has very ambiguous success criteria, right?

Host
Yeah but it sounds like what you’re saying is it’s the ability to define a problem with, with precision.

Alex Huras
Not necessarily with precision, but I think this, this generally comes down to like the holy grail of education, which is like critical thinking and like generic abstract problem-solving skills.

Host
Yeah. Okay.

Alex Huras
And so it’s easy for me to say, yeah, we need more people like that. I have no idea how to …

Host
… how to produce them.

Alex Huras
… how to produce that. Like that’s a huge education thing and maybe the learn to code initiatives or whatever actually do … like maybe that’s like the latent function of that is it puts kids in a situation where the incentives are aligned for them to actually be better at solving problems.

Host
Yeah, maybe it gets them in the door to where that is the work.

Alex Huras
Yeah. And so the big thing here is like you’re a small kid, you like playing with blocks or whatever. It’s like you start noticing that like arches are stable. That sort of thing will be with you for the rest of your life.

Host
Right.