
Leon Kuperman
Notas del episodio
El episodio 36 cuenta con la participación de Leon Kuperman, quien se ganó su reputación en el desarrollo de videojuegos gracias a las aventuras de texto al estilo Zork, y es cofundador y director técnico (CTO) de CAST AI. Anteriormente fue vicepresidente de productos de seguridad OCI en Oracle, y la experiencia profesional de Leon abarca empresas tecnológicas como IBM, Truition y HostedPCI. Fundó y se desempeñó como director técnico (CTO) de Zenedge, una empresa de seguridad que protege a grandes empresas con un WAF en la nube. Leon tiene más de 20 años de experiencia en gestión de productos, diseño de software y desarrollo, hasta la implementación en producción. Es un experto en computación en la nube, seguridad de aplicaciones web y el Estándar de Seguridad de Datos de la Industria de Tarjetas de Pago (PCI DSS), comercio electrónico y arquitectura de aplicaciones web. Y ahí es donde hablamos de su trayectoria en el desarrollo, sus ideas sobre cómo armar un equipo y un producto resilientes, y mucho más.
Capítulos
Transcripción: descargar (SRT)
Transcripción
Ep. 36 · Leon Kuperman · unas 16 009 palabras · marca el tiempo para ir ahíJohnUs independence day, happy July 4th to my fellow Americans. It's a bitter sweet holiday with what we've called out in my household. considering a giant step back for this country of freedom and opportunity with the us Supreme court, overturning Roe vs. Wade and making it illegal to choose to abort their unborn fetus. There's two sides to everything. And this one topic in particular, there seems to be no middle ground. It's either you protect life. 100%. Or you give the a hundred percent freedom of choice to the parent. I'm damn near borderline Buddhist in how I value every single life. But. My personal views or not, when it comes to the cost of the parents' wellbeing.
JohnI'm proud to say that both Amazon cats, employer, and epic games, my employer. Pay full costs for traveling and taking care of any medical needs to any employee who has to travel outside of their state From what I gather pretty much the entire west coast and Nevada AK the home of since city. Most of the Northeast with the exception of Pennsylvania, which was surprising to me. Michigan, Indiana, New Mexico and Colorado are all still. As far as I know. Legal and you have full choice on what you do. This is what happens when there's a key balance piece, like the notorious Ruth Bader Ginsburg, and she passes. Yeah, that was two years ago now. And then 45 appointed Amy Coney Barrett took her place.
JohnIt seems to me that that's how long it takes to see the effects of change com. You can't help. What's a speculate ahead at what's to come. And you just know that same-sex marriage has gotta be in their crosshairs. I'm guilty. I'm also watching Showtime series billions of this definitely affecting my thoughts, but while it's a fictional drama, It makes me wonder at the nature of politics and The influence that money has in addition to. Paying for lobbyists that influence policy and eventually. that's what gets topics up on a ballot to vote for? The same madness with gun laws and how many mass shootings it has taken to finally get a reform bill out of Congress in front of Biden.
JohnAnd that's going through, which increases background checks closes some loopholes and incentivizes states to individually pass their own. Further harsher laws. It's a small step, but that's the nature of policy and change in this country that I've seen in my lifetime. It's slow. but tiny steps make a difference. And also. And as we've seen, they can be overturned in the future. That's precious. I just want to call out to never take any of our freedoms for granted. And know that. We've got to keep pushing for change and progress because the second we are complacent. Bam. It's taken away. Big shout out to our neighbors, to the north happy Canada day.
JohnYou know, that's always a possibility too. I just straight up leaving the United States and going work for another country. Who has more progressive laws. And on a lighter note, July 4th is my father. He has his birthday Phillies complete on your Papi. I love you. Thank you for making it happen. And when mom Dukes and doing your part to bring me into this world and pass on your work ethic and hook me up with a solid education. Perhaps this audience will get to hear from you someday. No hit mom music. On episode 36 of out of play area. We fall out with Leon Cooperman, the CTO of cast AI. Who wrote his own Joerg text adventures got his start at IBM, founded a handful of companies, one purchased by Oracle and who is now working to look ahead and keep the cloud a better place to do business.
JohnWe talk about his passion in tech, coming from Ukraine to a family of engineers. Who's used on games and the ways that his company at cast AI is looking to make an impact. Please welcome coming to us from Toronto, Canada by way of Ukraine. Leon. Cooper moon. Let's fall the fuck out.
Catherinebienvenido Bienvenue Welcome to the out of play area podcast, a show by video game devs for game devs, where the guests open up one-on-one about their journey, their experiences, their views, and their ideas. No ads, no bullshit. Join us as we venture far out of the play area with your host seasoned game designer, John Diaz
JohnThe one thing I do not know about you, Leon, that I'm super curious is do you play games? If so, what were there, what was the first one that you played.
Leon Kwe won't have to edit much. Let's go. Oh man. You're taking me back. I think the first game that I ever played. Was a textual adventure game called Zork right where you literally, there was no graphics.
JohnAs soon as you said, text adventure, I was like, he's gonna say Zork
Leon KI was so fascinated by Zork. I bought this like shitty, manual, how to build a text based adventure game. I think when I was like 10 or 11 and all I had was an apple to eat. Like didn't have a lot of computing power. And I spent, I think the whole summer, writing text based adventures, it was like the most fun I've had, just coding for the fun of it. and that's really what drew me into computer science was gaming to begin.
JohnIt never tires me to hear that, right? Like this wide realm of computer science and most universities never, ever take advantage of that curiosity. Right. They kind of throw you down into logic and binary bits and, and things like this, well, I guess you start out with like basic input and output, but tell me more about some of the cases that you wrote yourself for your text adventure.
Leon KOh, it was a classic nerdy DMD thing, you know, it was like, and then I evolved it from, just plain text into a graphical interface of the dungeon itself. And so you have a little character that walks through the Dunge. I mean, gaming technology has evolved. Centuries, you know, since, I was playing around with code and then I continued to love that genre. So there's kind of two genres of games that I really love. I love a realtime strategy. So I think the first realtime strategy game I played was do June like probably in the nineties if I, if I had to guess. And then when my kids were growing up, we played age of empires, like together. That was kinda like a amazing bonding experience.
Leon KAnd then later I introduced him to sky and all of the kind, the prequels and like that's kind of been I really love the world that Bethe, the soft works has been able to create. I know it took a lot of money, effort and talent, but that game for me is something very special anyway.
JohnThat's a great call out. Yeah. Now they're folded under the Microsoft umbrella, right? So I think they're, they're gonna be steady
Leon KI really dug into kind of gaming from a machine learning perspective when the alpha star team put together their machine learning or artificial intelligence kind of version of their AlphaGo. You guys probably saw that documentary on Netflix which was unbelievable where AlphaGo was able to beat the world champion go, which is an extremely complex game.
JohnYes.
Leon KAnd then they stepped it up a level by playing StarCraft and training the models in a very similar kind of tournament style winner takes all. Game. And I was recently interviewing a, a data scientist machine learning prospect for, to hire. And we went really deep into that kind of algorithm, how they trained it. That for me is a fascinating field of machine learning because to be fair, they had to hamper the computer. In other words, they said, okay, well, like a human being can only have a specific field of you. They can't see the whole board. So let's limit the computer's field of you let's limit the number of actions per second to something, what an average human can keep up with.
Leon Kthey created all these restrictions and, like races. they still can beat the world's best individual humans, which is very cool.
JohnYeah, man. The, the field of AI is amazing. And, and I definitely look forward to touching more about how you're leveraging AI at your company cast AI. the latest one I saw this year is called w Kong AI. it's an evolution of, that Google space where, it just trains itself. It plays against itself, however many ludicrous numbers of hours to just figure out the strategy. And, and yet it's from a game development standpoint, we've always been trained as game designers that the best AI ends up being the best, second place, Like the best runner up. It kind of just gives you enough of an opening where you can still triumphant and feel like a champion, but.
JohnAnd these competitive e-sports environments, I can't imagine a better way to train, right? Like I'm, I'm sure a lot of the, the chess masters the best ones have incorporated like a lot of strategy that I'm sure a computer kind of showed them. They thought was kind of not optimal. And now they kind of whip these things out.
Leon KI think from a game development perspective, everyone looks at AI kind of as a, how do we get NPCs that are more human likely? So they're not just roaming on a very specific area in a, in a infinite loop, but they actually have daily lives and they, you know, they go to their jobs and they, you
JohnLittle like the Sims. Yep.
Leon KLike the stems. Yeah. And, and I think the closer that we get to kind of general AI, and we're still ways off, but I think the more realistic the non-player characters get in these game worlds.
Johneven the popular Hollywood takes one of my favorite ones, I don't know is like a, I remember Scarlet Johansen did the voice and it, you know, that was such a great casting option and it was walking Phoenix and yeah, it was just like, wow, they learn how to cater to your ego. Right. And, and that's what we, as men tend to love, right. Someone to cater to our ego.
Leon KJohn, have you heard of this, the touring test?
JohnYeah. Yeah. Isn't that the gentleman was named touring and it's like, this is the test at which you can no longer tell like, Hey, is this human or robot? Like, is that, is that kind of close?
Leon KYeah. So the scientist name was Alan touring. He was actually an heavy contributor into cracking Nazi codes in world war II. But he, he was a, a deep thinker in artificial intelligence before we could actually physically manifest computers to do what they do now. And yeah, he, he surmised this test where if you could put someone behind the kind of a wall and have a conversation, and you don't know if that conversation is a human or a robot or a computer, then that AI has been deemed to pass the turn test. And that's kind of the holy grail for perceived general artificial inte. Yeah. And these,
JohnIndeed. The realm of AI. Yeah. I mean like that's the buzzword. I think that's where all the funding tends to go these days. And I think that's what separates a lot of the. The big companies, right. Is how, how much they've had a chance to train their AI what is it using like captures now, right? Like that's how we're kind of going in and telling computers what images or what, and helping them categorize all that.
Leon KMost good. Machine learning folks can write an algorithm that will train against those. And, even to the point where, like, if you're asked to identify all of the railroads, that's not a very difficult machine learning problem to solve these days. Right? So captures are pretty weak. the separation of humans and robots on the internet is a very difficult problem. It's something that I spent actually years at my last company, trying to figure out and we were somewhat successful. But the problem is, is that when you're using robots as attackers in the cyber world, you're gonna be heavily incented to make those robots smarter and smarter and behave more and more like humans.
Johnof course like the more they can spoof and fool people, the more lucrative the back end I'm sure. And, and, and on the other end, the black market and security kind of goes step and step generally. Mm-hmm
Leon KThere's a lot of money to be made on the dark side. Right. So hence that's where the funding comes from. Everything from state sponsored all the way to, you know, basically organized crime. Yet the attackers are very well capitalized and run like efficient businesses in many contexts.
JohnI heard you discuss how much you enjoy RTSs and as I, get to speak more, you, Leon, I can have an appreciation for your logical mind and, being able to, control or direct or strategize an entire play field to some victory condition. I have to ask if you've ever considered getting into development, or has anyone ever approached you from the development side?
Leon Kof course it's been like a deep, hidden fantasy that I would never vocalize to anyone as they think I'm crazy. But yeah, I would love to have the free time to work on the creative end of game development. I think it's a young person's game for a lot of reasons. You have to have the stamina and energy. The coding hours are just insane. And I'm probably kind of beyond that level where I can do 16 hours a day of coding. But maybe there's a future where I can participate on at least on the creative. Game development side and maybe help with some of the difficult computer science problems that haven't yet been cracked.
JohnI definitely see an opportunity there, man, you're a company and, and the things you work on, being able to be a huge leverage point for a lot of us on the other end building, you know, whatever you wanna call it, right? Meta versus shared worlds, things like this. If I were to give you, the best of the best, right? The designers, artists, engineers, full team at your disposal, let's say two, three year runway funding and all that. What would you build?
Leon KThis is gonna sound crazy Joan, but I would try to build an immersive experience that breaks the player out of physically sitting in front of a computer and maybe even wearing in Oculus. Like, I think those constraints are severe, I believe that there's a gaming mechanism that puts you As close to a reality as possible. Let me give you an example. Like we were just at universal studios like maybe a month and a half ago in Florida. And I went on this Harry Potter experience and so these rides now incorporate a complete three dimensional screen. Like you're completely immersed in the screen wear. You are also on a physical ride. So at some point it's just visually tricking your mind into believing that you're taking a physical drop.
Leon KThey are incorporating heat. So there's blast like, you know, when that dragon blows fire, you feel it they're incorporating water and all these tactile elements into the writing experience. And why can't that be done for a gaming experience? as an example, why can't you really get onto a space shuttle as if light simulator and experience a star mission in the confines of like a stride Trek, like, you know, Rover or whatever it's. And, and I noticed that the ride experiences are getting there.
Johnmm-hmm
Leon KI think the there's room for the, for gaming to move in that physical direction as. Yes. \ John: I love the ambition. I love the, forward thinking, taking what we have on the horizon with AR and VR and the confines of our living rooms and, and moving in a step towards these massive theme parks. Right. Just that, tactile stimulus that we currently lack. Even uh, the smells is always a fun one to, to bring up. I like it. You got into computer science early. Was that something that came from your family or just the curiosity through games can you tell me more about how you stumbled into the com sci. I came from a, like a stem family, meaning my mom's electrical engineer by.
Leon KMy dad is a mechanic. My brother went into mechanical engineering. I mean, computer science didn't exist when they were going through their educations. Or I still remember, my brother was at Ryerson. It was the school here in Toronto. And he had a software class. I think it was Pascal. Was the programming
JohnYeah. That was like the first one. Yeah.
Leon KNot exactly, but very, it was way early. Right. And I remember like he did his homework on punch cards and I had to do like help him with his homework on the punch cards. So that was fascinating to me. And we had a computer at home. We could write the code, like on our apple, but the, the school needed it on these bunch cards. so engineering and math was always a part of the family. I think I got the entrepreneurship piece from the family for sure. And then I got the, the love and interest of science from my family. And then it just naturally evolved into computer science. When maybe when the first time I went to radio shack, I don't like, I, the first time I saw I saw a computer, I was like, just fascinated by it.
Leon KAnd then it becomes the 10,000 hour thing. Like if you're 8, 9, 10, and this new tech appears and you just dive into it for endless number of hours, you just can't help. But come out of it, being a subject matter expert, you know, 25 years later or whatever it's.
Johnyeah. You were even well ahead of the game, right? Like I'm sure everything that was thrown at you academically or in the curriculum you were already well, well, in front of.
Leon KYeah. I don't think I really studied until third year computer science, I think first and second year were part of years for me, cuz there was nothing really, of interest to deal with in the exams or in the assignments. But third year in operating systems, it started to get a little bit interesting. Started studying.
Johnokay. What, what was it about operating so simple? Was it kind of like memory manipulation or registries and things like that?
Leon Kit's a hands on course. You build an operating system through the semester. So you build a file system. You build a mini memory manager, you build a CPU slicer, and then it kind of comes all together by the end of the year.
Johnman, I, have to ask Leon, cuz I find this paradigm super interesting these days, right? Like you have the people that come from traditional engineering comp sci backgrounds, and then kind of this other wave of people that come up more vocational tech, like. Pure game development, right? Where it's just like day one, it's like here's unity or unreal. And then we're building a game. And a lot of us came from pure traditional logic and architecture and object oriented programming, and then found an outlet into like, okay, this is how we can build interactive experiences through here. Right. Curious what your thoughts are on having that deep, low level knowledge as opposed to the other end of it.
JohnRight? Like, Hey, just tell me what I need to know to, to do the thing, right. Or, Hey, I have an API that can do all the heavy lifting for me. I just need to know how to hook in
Leon KWhen I interview people for even software engineering positions, as an example, and I'm not gonna pick on the.net community, but like, like the, the Microsoft visual studio kind of.net, or even before C sharp, it was visual basic. And like, those are prepackaged universes of their own. You don't gotta know a lot, you know, to start and be productive and I always try to figure out what is the motivation of the person not to dig deeper. don't you wanna know how things work? Like are you content being in a bubble? How does unity do the things? It does? Where's the magic come. You don't have to build those algorithms every time, but at least understand the building blocks.
Leon KRight? So how many of vocational, like game developers know what a transistor is and know why a transistor changed the world of computers? Like why we couldn't do things before we discovered what a semiconductor was, why couldn't we build computing power that we have today? And I bet you, if you ask a lot of folks that just don't know,
JohnSure.
Leon Kbut that's lost tribal knowledge.
JohnYeah. Dangerous.
Leon Kif you ask that question to like, even a non-technical person, like if he has bill gates, who I don't think is a super technical guy, you know, he is a business person, how does the transistor work? I think he would be able to tell you. Right.
JohnYeah, he was
Leon Kyeah, so like, I think there's a whole bunch of travel knowledge that we've lost and it's sad, right. Because. If you have the curiosity and the background to dig into those things, those are the foundation of later ideas and the depth of your knowledge really makes a difference into how you think about the future.
John1000000000%. I love your frame of thinking there. And I want to emphasize that right. Is yeah. There's ways to get off to the races and get prototypes put together and things like that. But the underlying technology hasn't changed. Right. It's just kind of gotten stronger, more powerful. Right. But knowing what a transistor does, why does that give you so much capabilities? Another common one, I like to tell people is polygons and rendering triangles. Right? Why. Is that the chosen way that we go about drawing things to the screen? Why does a computer really understand three points in a space, right? Or, or the limits of a plane. And then being able to build on that, like, this is how lighting plays into that.
JohnAnd this is how things look more natural. And, and this is how you can optimize these graphics, renderers and GPUs to really pump out that blurred reality, right? Like now you see a lot of these simulations and you're like, man, that looks amazing. Is it just more powerful hardware or is it a better understanding at the lowest level that we can now take better advantage of it?
Leon KI, I think a lot of the advances that we have now are abstractions of APIs, right? So from the hardware all the way through to the gaming API, but you tend to lose resolution every time you go up a level in the stack, right? And someone else has made a technical decision for you at that lower level. Maybe those are great decisions and those should be the right decisions, but at least be aware. Yeah. I,
Johnabsolutely 100%, right? Like when it comes to these kind of second to second or to the frame, right games, we are bound to that frame, right? If we get 60 of them in a second you gotta optimize these things. And, and these worlds where now you're getting into the hundreds of players online anything that's off, you know, you can blame it on internet or, or latency or something like that. But if you can figure out the most minimum amount of data that you're transferring over the web, right. To kind of get those dependencies down the higher you can go, right? The farther you can push it as, as you.
Leon KI think when you start introducing multiplayer, it's a massively distributed hard computing science problem. Right? You have action. That's local and you have action. That's in the real gaming world, that's on the server. And how do you reconcile things as your connection legs? What are the elements of the network that you need to use in order to make sure that you can eventually do a full reconciliation of all of the universes that are playing simultaneously? It's a super hard computer science problem. It's a very difficult one to solve.
Johnyeah. And then, you know, it's not a secret game, developers we're expert, slight of hand manipulators. Right. We, we cheat as, as much as we can. We could as many quarters as we can just to always cater to the player, right? Like, Hey, what, what does it matter at the end of the day? What's the key thing players want to experience. And so here we are. We're talking about the web. We can't talk about the web without talking about cloud. tell me about your role where you are at today and your company.
Leon Kwe started cast AI just a couple years ago. so that's the company and it's solving a specific, in one sense, it's solving a very specific computer science problem around cloud economics. In another sense, it has a much broader vision. So let me explain the, the first step we took was in previous startups, we had failed pretty badly in cost management. Like everything else was great, but the cost side of the equation when we were deploying to AWS specifically, was a disaster and I could never get the kind of cost, a good sold bottle, you know, my last startup organized properly so that as we scaled customers into the thousands, the, the cost model would scale appropriately.
Johnwas that a slow burn of like, you know, month over month you're seeing these, these ticks? Or was it like a, easy to miss kind of thing. And then you're kind of paying before, you know it
Leon Kfrom when we started the last company, it was like a few thousand. We started a few thousand dollars a month for the basic in this structure. We're probably close to half a million bucks by the time we'd sold the business. So like multiple orders of magnitudes rising costs that was with, with reserved instances and all kinds of savings plans. Like we tried everything. And what saved us is, is the, the company that acquired the technology or the company was Oracle. Oracle had its own infrastructure. So cost of good sold. Wasn't a problem. We just moved everything into Oracle data centers and cost went virtually to zero. like we would've had to have solved that problem.
Leon KVery quickly, because as your market momentum grows, your customers get bigger. as your customers get bigger, the traffic increases and you're paying dearly for that traffic. If it's not well optimized, if the costs aren't well optimized. So the continuation of that thought process was cast. And I'm like, okay, I can't be the only guy who hasn't figured this out. Sure enough. I'm not because our customers all feel this pain a hundred percent. So we made a couple bets, John, we said, okay, number one, we think that applications, and maybe you can chime in on this, on the gaming set. I'm not sure this is a trend in gaming yet, but we believe that all like business applications are gonna be deployed in these things called containers or containerized units.
Leon KRight. So Docker containers and they'll be made up a set of microservices. So you'll have services that represent different functional. Parts of the system instead of a single monolithic application. And if that's true, then there needs to be an orchestration platform for all of these containers and their life cycles and how they live. And so we believe that system of record will be Kubernetes, which I can explain if, if listeners aren't familiar with it or some cousin of Kubernetes, but it will be that system that ends up being dominant in the industry, especially for business apps. And if those things are true and we're so short on human beings, then the logical conclusion was we need to have an autonomous system for all of this life cycle management.
Leon KWe can't have infrastructure managed by people anymore. So we make way too many human decisions. For infrastructure provisioning and the goal of cast is to take all of that away and to automate that through robotic process automation or through AI.
Johngive it to the machines. Heck yeah. That's what they do. Best crunch numbers. Tell me where I'm failing.
Leon KAbsolutely. so that's the premise for cast. And so we have three pillars of platform. And the first part of the platform that we went to market with is the cost optimization piece, which is my most acute pain previously. So the cost optimization piece basically allows our system as a SAS model software as a service to plug into a customer's Kubernetes cluster in any one of the three major clouds, Amazon, Google, or Azure, and first report on where the cost was, is so where the resources are wasted. And then given permission, it'll start taking action to optimize the cluster. And then within a few hours, you're running on a cluster that's 40, 50, 60% cheaper.
Leon KThan where you were before you started.
JohnI've seen Kubernetes, right? Like Docker came on the scene and it was just like, that's all you saw AWS prep pivoting to over the past handful of years. Right. And so the expectations is Kubernetes, everything. it seems like if you build it off of Kubernetes, you can run a report and then see your savings. Okay.
Leon Kand you may say later, Leon, why, why did you pick Kubernetes? Why not just generally optimize cloud instances, bare metal. And so it's too nebulous, John. Like, if, if you say, Hey, we'll, we'll just tackle everything. You're really tackling nothing. So I love Kubernetes because it's like a game board. it's like a StarCraft word. Like you've got actors, you've got actions that those actors can take. And there are rules of engagement. And if you follow those rules, you can teach a computer to follow those rules and you can teach a computer to optimize the outcome. Kubernetes is the game board in this context.
Johngeez man. You, you got me interested now, right? Like I, at the time I was like, okay not my cup. I'll let some backend engineers worry about this and tell me what I need to know. But now you got me interested, Leon, I'm gonna go check out what applications of Kubernetes may overlap with the types of things we do in games. you provide a report service that you point it to your cluster, and it'll tell you what you need to.
Leon KYeah. it's, it's a hundred percent free and always be free. And here's the reason why we do that. John is because if someone's getting, let's say weekly value, they're getting a weekly report, emailed to them about the state of their cluster and all their computers and how utilized they are and which teams are sucking up. All the resources. Get a really nice in depth report, including some best practices. I'm getting their data every 15 seconds. our agent is streaming the data, so it's not altruistic. Yeah, we want to give value and we want people to, to perceive that value and get it for free. But we're also getting something out of that equation, which is we're getting their data and we're getting it in the most unique perspective, across three clouds.
Leon KI don't think anyone. So like if we're in 10,000, a hundred thousand clusters, like, you know, within a year from an entrepreneur perspective, you try to build these moats mode around the castle. And that's our moat like data is our moat. It's gonna be hard to build those models. Even if Google wakes up tomorrow says, Hey, that guy, Leon had a good idea. Let's copy. It's gonna be hard because you don't have the data from the other clouds. Like you're, you don't have a holistic view. The way we're positioning ourselves to.
JohnGotcha. Gotcha. Gotcha. And I guess that's one big advantage as well, is that because it's Kubernetes, right. And all the clouds have hooked into have Kubernetes on their, on their infrastructure, your service kind of jumps across the all three of them and, and finds where the savings are to be had independent of club.
Leon Kindependent of cloud that most customers aren't in a place where they can leverage two, three clouds at the same time. it is just too futuristic for most customers of mine.
JohnYeah.
Leon KBut that will come. Like we already have an offering that allows you from a disaster recovery perspective to create a cluster that exists in two places simultaneously. it is. There's a couple of unnatural blockers. And I'll tell you what, they're one is the customer mindset. Ah, I just can't get my head around being in AWS and Google that's that's one. The second one is this nasty thing called egress costs and everyone hundred of your game developers feel this and pay for what is an egres costs. So clouds allow your data to stream in to the cloud at zero cost, $0 stream. As much as you want, as soon as data leaves the cloud, like let's say you're serving a video or an image or HTML or adjacent file.
Leon KYou pay you the account owner, pay for that data to leave. And you pay about 9 cents per gig, but now imagine you're running a massively multiplayer game. That's just got a little bit of data from every player streaming to the tunes of a, maybe a couple gigabytes an hour. All of a sudden your egres costs are gonna be your number one cost of running that game. Especially if you have a freemium model where your users may not be paying anything, they may just be having a good.
Johnyeah. Yeah. That's something that I wanted to touch on. I'm glad you brought it up is a lot of developers. I think if you're starting and you're looking into the space, Hey, you wanna be online? You're not gonna worry about. building your own little racks and, and, and maintaining the power and backing those up and everything, right. This is where the cloud came in it makes sense. Hey, go to the cloud, it's cheaper. You don't have to worry about it. But as you experience in your previous companies, when you get to that sweet problem of having a ton of players, it can happen overnight, right? Hits happen. Something, a tweet or some stream goes viral.
JohnAnd then all of a sudden, your game has hundreds of thousands of not millions of more players than you anticipated, because things are in the cloud, they will scale up. And like you said, so your costs available balloon and worse yet, if you are operating on ad revenue and the number of eyes that see your thing or the few people that are paying for. You could still very well get into a place where you haven't adjusted your numbers and your costs are more than what you're making. Right?
Leon KYeah. It's, I've seen horror stories of folks that have like, used like function as a service or some of the cloud databases where either a developer makes a mistake or there's just a balloon in traffic. And then the bill comes like, I mean, those are extreme cases, but like, And I don't wanna scare folks, like just put your safeguards in place to make sure you don't scale infinitely, but you're right. in my business cost of good sold is a for, for software as a service, that is the number one driver. So a gross margin, what we call gross margin. It's what you collect versus what you have to pay for those services, right? So it should be around 80 to 90% for a healthy software as a service business.
Leon KAnd then most of your money goes to acquiring new customers. And that's how you scale. That's how you grow. I'm assuming the, the game world has a very similar kind of macroeconomics where you want your cost to good soul to be fairly low. And you spend your ad revenue acquiring new gamers, right? Acquiring
JohnYeah. Yeah. That's what it is for sure, man.
Leon Kand if you are not smart about cloud economics, AWS Google or whatever can eat your lunch literally eat your lunch. And what's interesting is, and your listeners probably experience this when they first start you can enter these startup programs in any one of the three clouds and you'll get credits like we personally got a $300,000 credit, you know, I don't even know if I'm supposed to say it, but Google's not listening. Like we got a $300,000 credits, Google to start, guess what? We blew through that in a year. Right. And then what, like, right. And now what are you gonna do if you're not ready for the cloud, you're gonna get a very strong cloud economics lesson.
Johnindeed. Yeah. I mean, at, at the very minimum, right? There's like alarms and watches to set up right. To auto notify and, you know, make sure you're getting those, you you're paying someone, that's keeping an eye on those things. to avoid these surprises right. To at the very least have something very loud and in your face to let you know, Hey, this is crossed over whatever threshold you
Leon KJohn, I'm gonna tell you a little bit of a unkept secret about those alarms. they batch up overnight. So your bill gets calculated nightly. So if you start with a bad piece of code that does a huge overrun at 2:00 PM, you ain't gonna know about it until tomorrow.
JohnOh my gosh.
Leon KSo you have to calculate, you have to alarm on your usage metrics, not on your dollars. So what I mean by that is if, like, for example, you're in AWS and you're using CloudWatch set alarms and triggers. On your traffic, the traffic itself, right? Because you're gonna get built for that set alarms and triggers on the computers that you're using the, the servers, because those are the things that will ultimately yield to the lead to the build. But the curve, the what's called cost report usage and reporting module. It does a, it does a, a crappy batch thing overnight.
JohnOh man. That's a pro tip. That's well worth the price of admission right here. Do you set up your alarms on the usage? Not on the dollar amount and that you'll catch it much sooner than when it's too late. That's awesome.
Leon Klike our customers, like we, we emit something called Promeus scrapable metrics. These are just time series data that we we update in our platform every 15 seconds. So what our customers do is they connect their monitoring systems to our metrics and they, they continuously scrape them. And then within a minute or two you'll know exactly where the costs have gone. And then they use that to pin a team down and say, Hey guys, you had a bad deployment. You just balloon this cost 20 X or whatever. Let's roll it back.
JohnI'd like to talk a little bit about opportunities that you helped to point out. the very little cloud education I have. Right. my time over at Amazon was the early opportunities to be had regarding on demand instances to then reserve instances, right? Kind of you're paying in advance so they can sell that to you cheaper. And then at the very low end of the spectrum, Spot instances, right? Like the things that are like excess and too volatile for anyone to really take advantage of. That's why you kinda get a great deal on them. But the big secret is they were great for games. Right. Great. For these less than 10 minute sessions, 15 minute sessions, right.
JohnWhere hosts are constantly migrating, right. If you have a 50 on 50 thing, someone gets killed, the host jumps over to someone else. Right. Those what do you call it? Like, I don't know what some of them have like two minute warnings where it's like, Hey, this, this thing is going down. Right. Someone needs it now. So we're gonna take it from you. It was perfectly suited to game developers, right? Like spot instances. Mm-hmm and, and, and that's what, they have a service called game lift that helps kind of, you know, as a service around it. Gives you an opportunity to get versed in that again, it'll always pale in comparison to you spinning it up for yourself, right.
JohnAgain, empowering people to learn what they're doing so they could write it for themselves as opposed to kind of rely on AWS's service to, to tell you. Right. But what, what, what opportunities does cast leverage for, for people mm-hmm
Leon KSo I think game lift is a great bundled service. Obviously there's a pricing model and you pay for the extra features like, and it's got a whole bunch of stuff that's really tailored towards the gaming communities, kudos to those guys for packaging it, seeing the, the need and packaging it that way. And they do use like spot instances under the covers. So I, we have to, it would be interesting. Like I have to do some research to see how effective would it be to deploy a real time game. Inside of Kubernetes because Kubernetes does it's, it's pretty low latency network. Like, I mean, you basically get the bare network of the machines but there, you know, but there's some container runtime overhead, and you, you are run, you're not running on bare metal.
Leon KYou are running in a container image in a pod. So it would be interesting to see if someone has successfully. I, I suspect the answer is yes, because Kubernetes or K eights is the acronym has this really nice feature called HPA or horizontal pod auto skilling. What that means is, is as you get more players in the door, for example, like, as they're queuing up, the system will automatically spin instances of the game services that you need. And then as the Q empties, it'll automatically shut them down under the cover. the infrastructure that's running on can be elastic. It can scale up and scale down. Now. The, the AWS gaming service, the game lift service has that kind of auto scaling capability and they kind of tie that into spot instances.
Leon KSo I would say it's a specialized version for the gaming community. So what we do with spot instances, it's, it's a very interesting kind of Freakonomics problem. Let, let me explain what I mean, John. when we look for, to add a node into a cluster in Kubernetes, if the workload itself is tolerant of being interrupted and what I mean by that is like, if the process can get a signal that says, Hey, I need to kill you, but you're gonna get spun up somewhere else. So that's your kind of analog of moving like the, the, the game node from server a to server B. So if those workloads are, are I tolerant, we handle the life cycle of the instance for our customers.
Leon KYou mentioned a couple of other business models. So maybe before I go into deep into the spot instances, let me give you my 2 cents on like, kind of where I think those pricing models are and where a customer like, like a developer should be thinking about those pricing models on demand on demand was the first and probably the most transparent model. It pays you go, I need a computer for a couple hours. I rented $2 an hour. Cool. I paid four bucks. I'm done. And then customers starting to come to the cloud and saying, well, these things are getting really expensive. What do we do? So what did the cloud say? Very smart. Cuz it's a, it's a, it's a lock in protocol.
Leon KLike I want these customers locked in. Right? So exactly John. So they say, all right, well commit to us for two years, a year, three years, and we'll give you a 50% discount or 40% discount. But in early days of AWS, you were committing to a specific processor family. You were saying like I'm committing to the M four family, which is a specific Zion chip set under the covers. What, what happens in three years of your commitment? Well, according to Moore's law, we double our processing power of 18 months. Therefore we have our costs every 18 months. So you're committing to a pricing model. That's gonna be obsolete 18 months. So, so I look at reserved instances as an evil, it's a necessary evil, but it's an evil we, we want to try to eliminate and then.
Leon KSo the last kind of like what I consider the most market driven discount available to users is the spot instance. And if you can learn to deal with the chaos, if your application can handle the interruption, which we help with, then you're in the best position to to, to take advantage of the highest discounts. And so what are the problems with SW instance? The first one you mentioned already, John, which is two minutes, notice get off the machine, we're killing it, right? So that's a hard problem to deal with. That's where Kubernetes helps us, frankly, help the customer, cuz we've got some elegant ways of draining the workloads off the machine and getting them replaced into a different place within two minutes, by the way, only 7% of AWS users take advantage of bodies.
Leon KThis is for,
JohnWow.
Leon Kfor, for that exact reason, we play this other interesting parlor trick. When we were going to reserve capacity. So we, we have a copy of AWS's. We scrape all of the inventory and pricing every couple of minutes. And so we know when prices fluctuate because they do it's a marketplace when demand is high prices go up when demand is low prices go down, but they're different for different instance types. So you may have this really weird computer shape that no one ever uses. I'll give you an example. Inf one, these are called inferential processors. No one's ever heard of them. Fairly few people use them, but they're super cheap because no one ever uses them.
JohnAre they like for like computer learning or something?
Leon Kthey're used for inference. Right? So this, the, the second part of machine learning, like once you have a model built and you want to infer, you know, the hot dog, not hot dog problem on. So when you wanna infer hot dog, not hot dog, you use an inference process. So you could use that processor to speed things up and make it cheaper. The problem is no one uses them. It's an SDK, no one installs this SDK, etcetera, et cetera, but they're super cheap machines right now. So we see those spot instances are often a lot cheaper than the regular CFI family or an MFI family and AWS, most people order. So we have this Freakonomics nature, we'll get a computer with SSDs, the super fast network card, extra processors, we pay less money.
Leon KAnd then when that processor goes away, or that that family goes away, we just replace it with something else. And we're continuously doing that. So I guess the bottom line is you have to be pretty sophisticated about your kind of inventory management of what's available in the marketplace in order to make the best possible use of spot instances in the market. A
JohnGotcha. Gotcha. This makes total sense. Now, Leon, when What 10, you know, what's the patterns and what machines are not used and when, and, and how, and, and being able to kind of spot that well in advance of other people and, and being able to kind of give that, that savings right. To the, to the end customer.
Leon Khundred percent. That's the goal right. Is, is to get people's cloud costs down. And frankly, I think all the clouds will benefit from this is why I think they're pretty appreciative of our service. Then we can get everyone's cloud costs down that leaves more resources for our next generation to migrate more departments than every company start migrating. And we don't get this backlash of, Hey, this costing is too hard to figure out. Let's just go back to our own data centers. That was easy.
Johncast AI two years young, already making pretty good strides and then getting people's cloud build down. You mentioned that you have a few pillars. You have your eyes on as you move forward. Can you talk to us about the next the next venture in cast AI?
Leon KYeah, the, the, absolutely. I'll tell you about all three of them. I, I, I don't believe in like a secretive development. I mean, I'll tell, you know, like the best martial artist can, like, there's this guy in jujitsu, his name's Gordon, Ryan, he's the, you know, probably the number one grappler in the world right now. And he'll come to like these big tournaments against high level professional fighters. And he'll tell them, I'm gonna submit you this way. Once he gave the ref, like a little piece of paper said, open it after the match, it'll have the, the submission on it. So like, you know, say he's just super, so I, I'm not gonna, like,
JohnDo you
Leon Kpeople.
JohnDo you roll?
Leon KOh, I roll.
JohnOoh, fantastic, man. And, and so again, you know, is that a family thing? I, I your family hails from,
Leon Kthe Ukraine.
Johnfrom Ukraine. Okay. Okay. things are still terrible over there. If there's any way that you know of that listeners can help, please let us know.
Leon KJohn, you bring up a really good point. So, so I was born in Odessa, which is the Southern city and the Ukraine it's on the black sea. That's where the, kind of the Southern amphibious assault is, is kind of occurring. year, my business partner also born in Odessa. I don't have the accent cuz I was young. He does. So we talk about it all the time. But we are doing a program right now for engineers that currently living in the Ukraine that want to leave and are looking for a way to get out. And we are both kind of refugees from that former Soviet union system. So we understand how difficult it is to get out and get your feet on the ground. So we're offering folks we'll take you through a remote loop, meaning we'll interview you remotely.
Leon KWe'll, you know, we'll do the coding exercise with you and if we offer you a job we'll offer you relocation to ness, which is in Lithuania. It's we have a development center there. We're gonna pay for three months of housing. for the developer and the family, and we're gonna help take care of all the paperwork to the immigration, the visas, the lawyers, all of that is daunting and overwhelming. So we're gonna try to make it as easy as possible for folks and start their lives elsewhere. We're try to help there.
Johnfantastically on way to give back to the to the roots, man. Heck yeah.
Leon KYeah. It's, it's not a hundred percent altruistic. I mean, there, there's an element of that. I mean, I am gonna, I am gonna benefit directly, but I'm hoping that those, those folks will be, you know, we will be super loyal to them and hopefully they'll be super loyal to us, right?
JohnAwesome. Awesome. Thanks for sharing for that. And, and I'll definitely link in the show notes to the hiring page where people can click on and learn more about that.
Leon KYeah. Like if you guys can spread the word, that that would be huge, but I wanna get back to your, I wanna get back to your question, You asked me if I roll, I have a, a brown belt in Brazilian jujitsu. I've been rolling since 2009. I started in jujitsu by the way, a lot of tech folks in like in Seattle is a huge like community. Like when I was, when I was shadowing up the Seattle every week even like pre COVID, obviously, but there were some great little clubs out there. You guys have a 10th planet out there across the lake in Bellevue. There's some really good
JohnOkay.
Leon Kto roll. But going back to the three pillars, the second pillar is cybersecurity. So we are going to provide a set of Autonomous services to help secure customer clusters, make sure they're following all the best practices, make sure that their container images don't have vulnerabilities. And there's a whole suite of people do just a bad job of cybersecurity in the cloud in general. And I have my own theories as to why that is, but we're gonna try to help customers there. And the third pillar is disaster recovery and high availability. So how can you make sure that your application stays up? I'll give you an example. If you have your entire game hosted in AWS and that in a specific region, and that region goes down, your game's offline could be offline for two hours, three hours.
Leon KWhat does that mean? How long does it take for players to get back to you? That could be a two, three day ramp back time, depending like, cuz it's a chicken and egg, right? Like if you have no one to play with, you're gonna
JohnYeah. yeah, you can, I don't know. You can jump servers somewhere super far that kills your latency. Right. Which again is gonna ultimately make people drop off as well. Mm-hmm
Leon Khundred percent. So there's three pillars. The first one we're starting with is autonomous cost management. The second one is autonomous cybersecurity and the third one is disaster recovery, high availability.
Johngreat spaces that are gonna be, you know, there's no signs of slowdown anytime soon. Right? The more companies that come online and pretty much, I can't imagine the space that isn't right. Like they, they, these are things that they should be worrying about or should have a plan for. I don't care who you are really. Right? Like keeping costs, slow, keeping your information and data secure and then having backup recovery plans. If, and when things do go bad.
Leon KYeah. People ask me sometimes, Hey Leon, what's your Tam, which stands for total addressable market. It's kind of a. Measurement people use to decide whether your opportunity is worth investing in is the opportunity big enough. Right? And, and so my, you know, my answer is my, my total addressable market is the entire cloud, like layers. Literally everyone is gonna be deploying with containers. Everyone is gonna be using Kubernetes. So
JohnYeah.
Leon Kwe have a lot of people to help in the future.
JohnI know your role, right? You founded the company. I would love to know what a typical day looks like for you as do you go by founder, CTO, CEO. What do you go?
Leon KMy title on slack, I kid you not is chief plumber and sandwich maker.
JohnI haven't heard that one. Leon chief plumber and sandwich maker. I get the
Leon KWell, like, look, you're in a startup and someone clogs the toilet, someone's gonna unclog the toilet. And as often it's gonna be one of the founders is not. So you wear all hats in a startup. Right. And that's the cool part about being in a startup. You get to do everything sometimes it's distracting, but it's so immensely fulfilling, it's hard and most people fail and every venture is bound to fail. Like, so, like it's not something that, that same people usually try to do, but, but when you're there and you can get some inkling of success, it's like the greatest feeling. And John, yes. What my day is, most of my day is. Talking to customers, that's 80% of my day getting their feedback, helping them everybody is in support in our organization.
Leon KAnd that's the way I believe it has to be. So, customer connection is the most important thing for me. And then very rarely telling our teams what to do because I hired really smart people. Like they know what they gotta do, right? So it's basically walking the, making sure that the vision is, is being properly represented in our short term, medium term, long term roadmap. And then sometimes they'll bring me into a technical conversation where they could use some gray hair opinion. And I'm happy to chime in when that's the case. Like makes me feel.
JohnI haven't heard the term gray hair opinion. I, I got, I have the image right away, but what does that mean to you?
Leon KIt means a dude who's over I'm almost 50. Right? So like in software engineering community, that's, that's dinosaur status. Right? look, they don't let me code anymore, and these guys are way better than me anyway, but it's nice when we can have a good technical conversation and then I can bring up a learning from my past that is helpful to them in a go forward
JohnWhich IM sure happens. Happens regularly.
Leon Ksituation, probably more than I'm admitting, but yes.
JohnThere you go. Fantastic. You know, development, engineering, problem solving. You've seen your share. You've been in on many of those roles. You hire a bunch. What separates a great engineer from an average engineer. When you're looking at candidates, what helps I'm in there
Leon KIt's a really simple, I mean, it's a great profound question. My answer is gonna be tri simple. It's the reason that they're doing it. Right. So I start the conversation with don't. Tell me about your background. Tell me what draws you to computer science. What draws you to this field? Right. And if the answer was, well, I had a whole bunch of choices in university. I could have been a doctor or a lawyer, and I talked to my and said, oh, well, software engineering is super lucrative. That's the wrong reason. Yeah, it is lucrative. That's the worst reason. Cause last thing you wanna do. Is be sitting in front of the screen, super bored for a paycheck.
Leon Kwhy would anyone do that in a world of so much opportunity where you can just follow your passion or I don't understand why people do boring jobs.
JohnAgreed. I don't know what you've seen over the past couple of years, but I. This pandemic thing was a big wake up call, right? Where people, I don't know what happened, right. Were either forced to leave jobs or couldn't go into an office and therefore had the chance to finally take a step back and reassess and be like, wait a second. You know, like I can do other things or, or look at how much more happier I am now that I'm not going to an office. Right. Like, gee, you know, let me reassess my priorities and, and spending time with the family and things like that. So I'm with you, you know, if, if it hasn't hit you yet after this pandemic, why would you spend time doing something that doesn't bring you absolute satisfaction and joy?
Leon Kwith this gig economy, you can literally like, be anywhere it's like the, have you ever read the four hour work week?
JohnYes. Yeah. Tim.
Leon KTim. Ferris way before his time wrote that thing, but he is like, absolutely true. Like, you know, earn in us dollars and, and spend them in yen or where wherever he was at the time in Brazil. I don't remember where he was, but like that dude had. The right idea. I remember this really interesting story. It was this eCommerce company at the time, and I read the four hour work week and I sent it to my whole team and the chief operating officer, she was really like, Hmm, very skeptical older lady. And, and she, and she sent the CEO like, look what Leon is sending his team. They, he only wants him to work four hours a week. who was the best. I'm like, you completely missed the point of that,
JohnYeah, because there's so many great takeaways in there right. Of like learning to scale and, and automate and not be a bottleneck. Right. and hire people to do things. So that freeze up your time to make bigger impact elsewhere. Right. Like a lot. Those are the ones that I still remember anyway.
Leon KYeah, like super, that was a super impactful book. But, back to your question, like what, what do you try? I try to figure out where that person's passion. Lies. Like I know when I open a ID, integrated development environment for you guys, it might be unity for me. It's visual studio code, like, and I, and I create code that's that's artwork for me. And I don't want my artwork to be rushed. I hate it when I have to code against the deadline. Like sometimes the colors on have to take the time getting on the canvas. My wife super laughs at me when I talk about coding like art, but I truly believe it. It's like my it's my artistic expression. When I produce an algorithm that works sufficiently, it's like, no, no better feeling.
Leon KAnd I need to see that in a candidate. I need to see some similar spark of passion and they'll U it'll usually come out when they tell me about a hard algorithmic problem that they've solved or a hard distributed computing problem that they solve, or a hard machine learning problem that they solve their eyes light up, that spark shows. And then I understand that there's a culture fit.
Johna good portion of this show is spent educating and preparing other people to enter development, and things to be prepared for things to keep in mind. And, and that's a great one to call out is Know why this role, while you're applying to this company, why you do this thing be prepared to talk about either the hardest thing you had to solve or the thing you're most proud of having to solve. Right. I think those are common ones.
Leon KYeah, and those are standard interviewing questions. And I used to think that it was like having a formal computer science education was a big deal for me because like, you get to like the nuts and bolts and you go through all of that, the grind of learning, the lower level, like we talked about it early on, right. I interviewed a guy maybe two weeks ago, great guy. And we hit it off on all the technical side and on the culture fit. And I looked at his resume and I'm like you know, is, I don't, I'm not gonna say his name, but like, I'll just use John. Hey John, where did you get your computer science education? He's like, no, and I'm self taught.
Leon KI never went to college. I'm like, fantastic. Like I could carry a super technical conversation with you. And we really geeked out on a bunch of these cloud concept. And you never went to school for it, like congrats for saving four years of your life. Cuz I couldn't do it. Right. So, you know, hats off to those guys who can teach themselves. Cause it's a super impressive skill.
JohnI love it too. And, and, kudos to you for not catching that on the resume and instantly kind of throwing it aside, right? Oh, university's not even in here. I don't recognize it. Okay. Pass or next. Yeah.
Leon KAnd then at the same token, I've yesterday I interviewed a PhD student and like 15 minutes of this is a conversation. I'm like, dude, this is not going well, this is not gonna be a, like a call like super smart academically, but didn't have the drive. Didn't have the passion. Yeah,
Johnthe few graduate students I know, depending on where they are, right. Are either ready to be done or some people just fit better in the academic world, I guess.
Leon Kthat you nailed it. Just, just a better fit.
JohnI have yet to have direct reports. I'm in the IC route, I think one day and the not too distant future, I can see myself going into management. You had this amazing insight and I want you to share it here. It's like, how do you gauge your team, right? Or the people that you report to Because I think a lot of people fall into the trap or managers fall into the trap of keeping the status quo, keeping everybody happy. And I don't think that's the best way to approach the team for long term success. And, few other places have built in to kind of constantly keep people sharp and making sure they have, stat ranking or whatever.
Leon KYeah. performing. Yeah. It's like through performance reviews, right? And like, I think the extreme example of some old school thinking is Jack Welsh is the CEO of G he had a, a rule. every year, the bottom 10% of the sales team gets fired. matter how well they.
Johnthat's it. That's it.
Leon Kand I don't believe in that, but that is an extreme case of performance manager. But I think the reference you were talking to is if you are unsure about a team member, if you're unsure about someone on your team, whether they're performing or not, there's a instant litmus test. You can ask yourself to get it's a gut check. And the gut check is if that guy or girl walked into your office tomorrow and said, I quit, what would your instant emotional reaction be? Would it be one of relief or would it be one of anxiety? And then the answer is one of relief. It means you probably have to work that person into a performance improvement plan. And then if that performance doesn't improve, then you gotta let, 'em go.
Leon KCause it's not fair. The rest of the team, if you don't have people equally pulling their weight and putting in the same sweat, equity and labor. labor of love, then you're gonna have other team members that get disgruntled and say, ah, fuck it. Like if Leon doesn't care, why am I trying so hard? And then everyone's performance suffers and that's a rotten apple in a barrel.
JohnMm-hmm
Leon KIf you're gonna manage performance, you gotta do it proactively. I have a policy of doing performance reviews often and quickly. Meaning every six months we do a review of everybody's performance on the team. And we don't only do that, review to give people a pat on the back, we do that to compensate them. So bonus stock options, pay, bump, whatever that is, is like the recognition of good performance has to come with a reward. And then the opposite is true. Like if you're not doing well, we're gonna figure out what the plan is to get you perform. And many people turn around and do really well after that. But some people don't and you're actually continuously improving your team that way.
Leon KAnd then just another, like my tidbit on performance, I don't believe in spreading like if you had like 10,000 bucks in budget to give, pay raises this quarter. Right. I think the worst thing you can do is just kind of spread that around equally across the team that's kind of the worst version of communism for me, like the, the people who work the hardest and have shown results deserve the game. That doesn't mean next six months or the next quarter, other folks don't step up and, and, and get rewarded. it's like the term peanut butter spreading peanut butter. It's just the worst approach. Right. And I, I often red flag managers that do that.
JohnYeah, let that be a, a strong takeaway it's a great conversation piece as well, it's the same problem of you have that you could call it an underperformer, but really it's like not performing to the same degree as everyone else on the team and kind of dragging the team down is the same way of saying like, Hey, as a team, we'll all bask in the gain or the excess of the rewards, but it can be much more inspirational motivating that you do recognize the standout performers. Right. Cause then everybody has a bar to reach towards to be like, oh, you know, look at so. And so they did this, I saw. Their contributions. I saw them putting in the time or fixing these really big problems when they didn't have to.
JohnAnd I see them get recognized. Good for them. I wanna see what that feels like. I want to get there. I wanna be in the, and the should come next performance. Right. And I bet you, that's gonna prompt a one-on-one and be like, Hey man, you know, if, if it hasn't already, if, if for whatever reason, right. They haven't realized that to, to come at your door and be like, how can I get there? Right. Like, or, or, Hey, I see these things that I'm working on. Will this move the needle? Right? Is this the best way to spend my time or elsewhere? Right. Like, I think it's just all positive conversations to be had after something like that.
Leon Kyou nailed it. And at least it's out in the open. Like everyone knows exactly what
Johnmm-hmm yeah. It's not invisible that kills me. I, I know there's a popular conversation out there in the world of like pay transparency. But more than that, right. Is, is transparency of like company values and what do we reward and what are we looking for? Right. Like, Hey, we hired you because you fit the culture and, we believe you can deliver these things. Right. And it's, so if, if everybody's kind of on, at the same benchmark, right. And then there's people that can still overachieve and, and, and jump across the room of everybody's a genius, I think that just makes everybody sharper and keeps everybody pushing. you've been in tech for a while and, and I've been in it for a bit and.
JohnThere's no end there's shortage of new opportunities and new things to conquer or to learn about. And it's exciting, man. It always surprises me to run into someone that tells me, oh, I've I've done it all. Or I'm not interested in something, you know, I don't just, it loses that, that zest or that spark to keep learning or keep trying new things.
Leon KYeah. It's actually during interviews when I'm kind of interviewing for culture fit, you know, I like to ask why like, Hey, you designed the system this way. Why? Right. Sometimes the answer will be, I don't know the decision was made before me, or I don't know. My manager made that decision or the senior engineer made the position. I'm like, Does this sound like a good idea to you? Doesn't sound like a good idea to me. So why wouldn't you ask the question and if that individual didn't ask the question that tells me something about their character, I don't need. Yes, man. I want people to be passionate about their positions. If they think something is wrong, I want 'em to speak up.
JohnYeah,
Leon Kthey haven't spoken up in the past. The chances have them speaking up on the, you know, the probability that we have a good culture fit is lower.
Johntotally. This is something that I've come across over the past month or so. That was kind of surprising to me. I think I fall into the quadrant of agreeable, you know, agreeable as opposed to disagreeable. And there's a metric there that says that the people that tend to rise up the ladder or in positions of power or management or whatever. Are those disagreeable people. Right. And you know, there's definitely levels of tact in how you voice your, disagreements something that, that I realize I definitely have room to ask why or challenge when something doesn't make sense to me, as opposed to, like you said, yes, agree. Okay, this is the decision.
JohnCool. I'll rock with it.
Leon KI'm the opposite, John. I'm a total shit disturb. Like, I, I, I rattled the cage. Like if something doesn't make sense to me, so we have this pretty good cultural core value I didn't invent it. It kind of came from Amazon and AWS, but it's like the basic essences disagree and commit like you and I can have like a vehement disagreement about a tech decision, right? Like, we'll hear each other out. Someone's gotta break that tie either. We agree. Like we pick a direction somehow we pick a direction and we go, and then there's no backstabbing. We both go a hundred percent in. And I, I just had this with my CEO. Like I, we, we had a nice blowout yelled at each other for an hour.
Leon KAnd then at the end of that, sat down, had a nice glass of wine. I said, all right, Erie, here are all the reasons why I disagree with this decision, but if you're gonna make it, you're the CEO of the company. I'm a hundred percent behind you and I'm gonna make that decision successful. That's the only way to roll in my opinion. Like you can't bottle it all up, but you also can't have sabotage in the company. You, gotta stand behind a decision a hundred percent.
JohnI appreciate that. I appreciate you reminding me of those Amazon leadership principles. They are, there was 14 when I was there and I think they added two after I left
Leon Kyear were you there
John2018 to 2020. yeah. Yeah. Yeah. And so that was I think since then they've added two along the lines of like good citizen, right? Like, I think like long term sustainability or like green energy or something like that. I, I I'd have to look them up. Ooh,
Leon Kso there are way too many of them, like I have four core principles and those are the only things that we kind of guide our decisions. Customer obsession, cust you know, figure out the customer journey. All else we'll follow. The second one is taking ownership and leading by example, the third one is developing yourself, your team and hiring the best people around you. And the fourth is embracing change. Understanding that change is constantly occurring, have respectful discord, respectful conversations, but once you get to a point, disagree and commit. Yeah.
JohnThose are solid custom obsession, ownership, hire and develop the best and disagree and commit. That's one that I can do some more reps on. I'm good at committing. I gotta be better about voicing my disagreements when I have them. Right. Like I think everybody can identify when they get the little knots in their guts something that makes them kind of bat an eye. Of like not quite aligned. And that could be for a few reasons, right. There could be, you may not be understanding something. There could be a disconnect but the worst thing you could do and I'm guilty of it is not raising the question. Right. Hey, Hey, hold on. Can we step back? What do you mean by that?
JohnOr, you know, what does that entail? Things like that.
Leon KIn, my tech discussions with my team, like if we're getting to kind of a critical. I think they're being too quiet. I pick the scab, like what's wrong with you guys is you don't have your coffee. Like if I'm right, we're definitely in the wrong path. You know, like I need you guys to speak up. I pick the scab and I often get the reaction I need out of that. Like, you know, maybe there's a case where everything is logically laid out and there's no reason to have that debate, but often I know what the other side of the argument is. Right. it's not that I wanted argument. I want these guys to learn to, like, if I get hit by a bus, They need to be critical thinkers each and every one of them.
Leon KAnd like, that's the legacy, right? The making sure that, that if I get hit by a bus or if I go, if I get ship wrecked, as for our work week, the litmus test is you come back and your company's running, but thriving, you know, that you've done the right thing in terms of your organization.
JohnHeck yeah. I, I forget what I read. It was some book that was referred to me about leadership or whatever, and, and it's kind of like, you can go down the history of companies that were built to succeed after their founder or CEO stepped down or companies that crash and burn, right. As soon as the one person, wasn't there to keep the ship together, it all crashed and burned. So that's it right? Like. Build the culture that can sustain itself after you're gone or not available, or whatever, work
Leon KAbsolutely. Yeah. I got this great piece of career advice early on in my life. A good friend of mine, rip Simpson. He, bestowed on me. He runs a great little company called snow commerce right now. They're just killing it. But he said to me work to work yourself out of a job, and you'll never be out of a.
Johnto work yourself off, out of a job. And you will never be out of a job. Yeah.
Leon KWhich is basically constantly get yourself out of the path of execution. But as that happens, the scope of responsibility increases, your company grows customer side, you know, everything is kind of ballooning around you and you have to plan for that next level of scale as a leader,
Johnthat's a great blueprint to start out, right. If you're kind of wondering like, Hey, where can I go? How do I grow? Or, Hey, I have to write a three year plan five year plan. That's a great starting point. If you're not already thinking that way to help you kind of flesh that, pR FAQ of your life for something,
Leon KAnyone that chose me a three year plan and I'll call bullshit before I read it, nobody knows what three years looks like in the future.
JohnTell me about it. Yeah. I struggle. I was asked at my last job at EA. Hey, write a three year plan. And I was pretty good about writing kind of a one year plan, right. Of like, Hey, a year from now, this is what I hope to have contributed to, or be working on the, the three year, the five year was, I don't know why it was daunting. Right. Like even even think about it, like working backwards. Right. If we're here, some hit game on this technology using these systems and then working backwards, even that was like too daunting, right. Or unrealistic, right. It just becomes fluff. Like the whole middle part of that sandwich just becomes fluff that it's not gonna happen.
Leon KWhat was your favorite gaming environment to work in? Like as a company?
Johnyou're gonna put me on the spot, but in the spirit of. that's been an easy one. I've only been what, six places, epic being the current one. And, and I'm too young at epic, right? I've only been there what, like three months, if that, and I'm still trying to get my feet and foundation underneath me. Awesome culture, super smart people learning a lot. Fortnight's a, an insane tanker going at like Lamborghini speeds. But prior to epic, I was at EA and EA is one of the oldest game developers in the business, right. Still going today and they're public and all that. And if you were on the outside listening, there's nothing but terror stories of like EA being this conglomerate and corporate behemoth and like no soul and, no creativity, blah, blah, blah.
JohnBut on the inside, I had a blast. I had a blast. I, I, I, I really benefited in a way that they've learned where all these other brand new startups and, and game developers are still trying to figure out how to get to this like sustainable place of game development.
Leon Kbelieve it, I, I had a similar experience at Oracle. Like when I joined Oracle through the acquisition, you were like, oh, they're gonna suck your soul. It's gonna be the death of you creatively. And I was like worried. And like, I, I got there and it was a little weird at first cause never been at a big company. And I'm sure a lot of people didn't like the fact that I was asking, why do we do things this way? So many times. Hey, this sounds ridiculous. Why are we doing this? and then some of those things I changed I was like privileged, right. I had like a vice president position. So like, I, had a little bit of like say, and then I got a chance to like really work through the system and build some new products.
Leon KI got the chance to meet Larry E that dude is like super sharp or I was blown away and I was riffing on some, some next level ideas with him. And we were like talking about how Amazon would react and so forth. And I, I had like, nothing, but like an amazing, like all of politics and garbage aside. Sure. That that's there in every big company. I'm sure. I mean, I haven't been to a lot of big companies, but I can imagine it's like that everywhere, but the experience was like, cool. Like I wouldn't trade that experience for anything because it like gave me a whole new dimension to, what a corporate life actually looks. Being able to exercise your creativity, even inside of a big company and not being that cog in the wheel.
JohnIBM was kind of that first big business role that you had. And then from then on you went full entrepreneur.
Leon KYeah. I, I was recruited to IBM. Like there were two positions that, of like 300 applicants. I have no idea how I made it but I got recruited to IBM like to Toronto laboratory in a hardcore engineering position. As an intern and I spent 18 months there, it's kind of boring it sometimes. But like, I, steered my way into a team that was building like this.
Johnyeah.
Leon KECommerce platform before eCommerce existed. And like, I was like, I was one of the, like six engineers on that team. And it was like destined to fail, but it succeeded. And we launched like this company LL B, which is like a big cataloger in the us online before anybody. Like, it was very few people doing transactions online and all of a sudden that whole division blew up. It got funding from inside. I remember ner. So like, it was also a very unusual experience in that I was 19 years old and I got a chance to experience like a startup within a massive company. Right. And the entrepreneurial spirit took over. As soon as I got it started getting guardrails like, oh, you can't do this or you can't do that.
Leon KI'm like, yeah, I'm outta here. I'll just do my own thing.
Johnthat's usually the nail and the coffin or the sign, right. Of like, I don't know. What do you call it? Bureaucracy bureaucracy for the sake of bureaucracy, right? Like, Hey, this is the process. And you have to go through these checks and balances to make a big change, as opposed to, Hey, this could really help. And I'm interested in passionate about it. I'm gonna take ownership, you know, let me, let me do this. Right. I'm I'm making the same money anyway. I'm just gonna bring you more money. And people just stop that because it's not by the book or something.
Leon KYeah. I remember like I had a mentor at IBM. Like the guys like still keep in touch with him. Like he, he was the lab director and big executive at IBM. And like, he, when I wanted to leave, he was like, oh, you're leaving too early. Just hang out. I'll introduce you to all the right BCS in the I'm like, dude, I already got the fun thing I'm out, but it was interesting. I was more of a shit disturber than I think I realized, like, for example, I had like a fish tank in the office that was, there was no pet policy in the office. Like I just brought a fish tank one uh, weekend. Like my wife bought it for me as a present. And, and then like, on the exit interview, the HR person was like, you don't not have any idea how much trouble that fish tank caught.
Leon KLike we have meetings about like, whether we could let you keep your fish tank or not like you see, well, you're just validating my, my move here. Well,
JohnYeah. Yeah. We had meetings about something that doesn't affect anybody, but you and you didn't know about it. Yeah. so good exposure at IBM and enough of a, of a motivator to go do your own thing. So a vital part in the path. Yeah.
Leon KJohn, have you ever read dad poor.
JohnI think I read that in my like mid twenties.
Leon KYeah, me too. But there was, there was one lesson that stuck. Like, I, I don't know if I agree with everything that, that guy says, like Robert, Robert KAK his name, but the book is a must read for like, if you're just getting outta college, you don't know anything about Marty,
JohnThat's the.
Leon Kgo read the book, it's gonna save you a lot of credit card pain. Right. But one thing he said, like, you know, when you come outta school, you don't know what problems to solve. Right? Like you have no idea what the issues in the industry are. So go work for someone, go work for zero. So it doesn't matter. Go work for someone, not because for a paycheck, go work for someone to understand the industry that you're, that you're interested and passionate about, where the actual friction and problems are and that's what I did at IBM. I discovered eCommerce. I'm like. This is bad the way eCommerce is being delivered. And the shrimp wrap software is terrible.
Leon KIt needs to be in the cloud on demand, you know, and I went and did that, but like, I didn't have any idea of how to solve it until I felt an actual friction point.
JohnI like to put it as like, go get your real world education on someone else's dime, right? Like go into, see how. Companies can be better by do, going to someone else's company. Right. And okay. If I build my own company, this is how I would do it differently.
Leon KThat's basically it. Right. And it may not be that you build when I build my own company, but it's like, if I built a product to solve this problem, I would do it differently. Start at the product level or the feature level.
JohnLeon, I look up at the time and the time has flown the F by, but if you are ready to get into what I like to call the final round, you know, I'm a big fighting game player and the final round consists of short questions. Whatever comes to mind and we blow through them.
Leon KPerfect. Let's do it.
JohnCan you tell me the last game that you or your children finished saw the credits role?
Leon KPlanet coaster with my nine year old daughter. It's a SIM, it's a SIM for theme parks, amusement parks.
JohnOh, okay. Okay. Rollercoaster tycoon type thing.
Leon KYeah. But more real.
JohnWhat is the last book you read?
Leon KOh, I'm constantly reading. So I'm currently reading the wheel of time series. it's a massive epic. I think I'm on book six but it's called crown of swords.
JohnWhat's the setting.
Leon KSo have you ever heard of the wheel of time? It's Robert Jordan he's passed away now, but he's like this epic author. This is Amazon has has released it as as a series now. On Amazon prime.
JohnAh, okay.
Leon KSo it's based on this kind of Eastern belief. That time is a circle and that ages come and go, but basically history repeats itself and it takes you through kind of this epic good versus evil. it's very similar to kind of the genre of Lord of the rings or even game of Thrones. it is. And that's my kind of number one, non fiction genres. I love reading fantasy.
JohnYeah. something that's not the day to day, which you see around you.
Leon KYeah. So I have, I, I always in parallel one book, either audio or reading, one book that's nonfiction in one book that's fiction. And I kind of read two in parallel.
JohnI was going to ask what's your format of choice or what, device do you read on.
Leon KI listen on audible and I read on Kindle.
JohnYeah, that ePaper, that's my, my go to as well.
Leon KI did want give a shout out to the non-fiction book that I'm reading. So there's Ray Dalio is a famous kind of investor, hedge fund manager. He recently published the book and an accompanying kind of YouTube. Educational video that takes you through the whole book in about an hour, it's called the changing world order is the name of the book. Great nonfiction read. And it's basically about the rise and fall of empires default currencies, where he thinks we are in terms of where the us empire is as its status quo, like leader in the world in terms of it, it being the, the peg currency of choice.
JohnThank you for that call out. I had no idea. There was a new Ray Dahlia out there, and this topic has been at my forefront. Anyway, as I'm just digging into currency world bank crypto, the fact that we're not backed by gold things like this, right? So this is really on brand for where my thinking is these days. So thank you for that call. Hmm done
Leon KYou know, who does a really good job of explaining like the philosophy of money, how it was created is Sapiens. A short history of mankind. And I always mispronounce the author's name. His name is Yuval if you haven't read sapiens, it's like a must read classic
JohnIf you could have three dinner guests that are alive, would they. Yeah.
Leon Kthere's so many people that I would love to talk to. I think Hawkings would be one that I, that I would love to speak to Albert Einstein would be one that I would love to speak to. And then from Russian history there's a character named Ross Putin. It was like a priest. And I would love to understand what was going through that crazy dude's mind. I would love to speak to iron Rand. it,
Johnname?
Leon Kshe wrote the fountain head and Atlas shrugged. She's.
JohnAlice shrug. There we
Leon KYeah. She like her philosophies. These very kind of libertarian philosophies are a lot cornerstone, a lot of beliefs of the tea party and the Republican party. But regardless, I think they've all been warp, but like if you have ever read Atlas shrugged, it is quite an interesting piece of perspective on, on world and Liberty.
Johnanother good one. Boom. I give, I'll give you that. Fourth one.
Leon Kyou, man. I know I broke the rules.
JohnAll good. I love it, man. Shit disturbing and pushing the boundaries. That's what we're about on here from what you see of the industry. What is one of the things that you think we can do better?
Leon KGaining industry or tech in general.
JohnThe one that you're most familiar with?
Leon KYeah. So I'm most familiar with cloud and I think that we can do a lot better in letting customers of the cloud move their data around without being locked in. So everyone is busy trying to lock everybody out of and gain a competitive advantage instead of having open interoperability and just, letting customers move freely. And guys, there's plenty of business, like stop it with this nonsense and let people consume these cloud resources naturally without artificial vendor lock.
JohnI like that one. Right? It's our data. We should have free unrestricted access to it and it should not be hard for me to do what I want with it. I, if that is to migrate it somewhere else, right?
Leon KYeah. And look, I charge a fee charge, a reasonable fee, but don't charge 30 times the cost to move data. That's just ridiculous. And that's the number it's, I've done the math. It's 30 X.
John30 X, geez. And that's so arbitrary. Right. That's like, so made up to be like, Hey, this is how we're gonna.
Leon KI, I don't think it's arbitrary. I think it's deliberate. It's like we, you know, it's hotel, California, check in anytime you like, but yeah, you're never leav.
JohnAll right, Leon. We are at the final question. and that is if you had a good time falling out of the play area, is there anyone that you would nominate to fall out of the play area behind you? That could be a mentor colleague, someone you think has an interesting, compelling story for anyone of the art listeners?
Leon Kthat's a great question. I have a great, I, I don't know if she, she would do it, but there's a someone that, that I took up as a mentor while I was at Oracle, I was introduced to her name is Patty Azzarello. I can make an introduction to you on LinkedIn,
JohnThat'd be fantastic. Yeah,
Leon KYeah, she, she is a corporate coach. She was like a former marketing executive at HP, and she's done a whole bunch of, she's got, she's written two books one's called move. And the other's called rise on how to deal with all of the pressing priorities of, of kind of delivering and how to juggle those priorities and ensure that you're like moving forward as opposed to spinning your wheels. So to speak. She was a fantastic mentor for me. I would love if you can, if you can grab her, she would be a great guest.
JohnI appreciate an introduction and all we can do is try
Leon KAbsolutely.
JohnLeon, we made it, made it to the end. Thank you immensely. I learned a lot. I can't wait for my listeners to get a load of this and learn about the things, cast their eyes up to learn about the different ways that cloud is doing things. And some of the things they should consider for, their business, right. It could be just the way that they handle their infrastructure internally or for their players. And on top of that, there's just a lot of cool ideas around the space of development in general. And most importantly, some of the things cast AI is doing for employment for the people in Ukraine.
Leon KHundred percent and we'll, get those links over to you, John, so you can link 'em up.
Johnabsolutely. Do you have any last words before we.
Leon KYeah. John, great questions. And I love exploring the gaming space with you. And thank you for your insight.
JohnFantastic Leon. Hey man, I'll stay in touch for sure. Seeing what you guys do as you move forward in making the cloud a more affordable, safer, and
Leon Kplace to do business.
Johnbetter place to, there we go. There we go. Awesome. We will stay in touch. Take care.
Leon KThank you, judge.
JohnTalk about falling out of the play area. Technically Leon has never shipped a game, which is the key criteria of all the guests that come on the show. But he has developed games and has shipped products that game devs can use. Developers or developers. A few parts of resonated with me. Six month performance reviews, keeping that conversation more frequent. And transparent and rewarding, outstanding contributions to the business. Before Amazon, I witnessed what happens all too often as studios who are fighting to finish and ship a game and drop everything that doesn't contribute to that. And are super quick to sacrifice employee development and investment.
JohnYou've all heard me preach about by monthly, if not weekly, one-on-ones both with your manager, your reports, and also your skip level. Another key piece of that equation is the biannual, if not quarterly performance review conversation. These naturally extended to one another where you're constantly having that conversation. Of the impact you either making, not making or should be making to get you to the next level of where you can better contribute to the business and the team. How many of y'all are up on your cloud spending and keeping a close, careful eye on that. I don't know about you, but I've never been a fan of not keeping my spin in check and making sure if there are any savings to be had that I take advantage of them.
JohnI think my partner in life, Catherine, where, yeah, I'm much healthier financially. And I know it was 10 years ago, living in California. And a huge part of that is keeping tabs on interest I'm paying, taking advantage of all bundles, subscriptions, bulk or wholesale purchasing that I can make the save a dollar. So with that, there's a promo link for out of play area listeners in the show notes To claim a cast AI offer specifically for out of play area listeners. I also decided to look into like, Hey, do any people in games actually use Kubernetes for their backend? And I found an interesting article that specifically talks about unity game backends built on Kubernetes.
JohnSo curious if, uh, that helps you at all, any mileage you get out of that, please let me know super interested. As far as I know, when I was working at Amazon and I never really knew of anybody. Strictly deploying through Kubernetes. On episode 37 of the game developers podcast out of play area. And we'll sit down with Chris Cole, a dear friend, and a senior technical artists currently working for Amazon game studios and whom first started alongside me back in 2007, a full year before the Dawn of the Marvel cinematic universe and Chris Nolan's dark Knight. A time where you may be recall everybody singing or listening to Rhianna's umbrella. Chris Cole and I both started at mid way, Austin.
JohnThat episode is set to debut in a couple of weeks on a Monday, July the 18th. make sure to follow us so that you don't miss out on that episode. Thank you for listening, Deb. If you found this episode informative, I ask that you pay a link forward to a developer to help grow our listener. If you're a game developer with a story you think could help a fellow dev out, please go to out of play area.com and click on the Calendly link at the top to meet up, please make sure you get approval from your manager or studios, PR HR team beforehand. Out of play area, the game developers, podcasts releases, new episodes every other Monday on all the major players, including Spotify, apple, and Google.
JohnPlease make sure to follow us, to see what developer falls out of the play area. Next time. I'm your host John Diaz until next time devs stay strong. Stay true. Stay dangerous
Texto sin formato: outofplayarea.com/podcast/ep-36.txt
