What I’ve learned after 1.5 years as a Junior Software Infrastructure Engineer
Kensho's Rayshard Thompson reflects on his first 18 months in software infrastructure, and the lessons he'd pass on to anyone just starting out.
Goodbye bubble sort
So, I graduated college with a BA in Mathematics and a concentration in Computer Science in 2021, with my career starting at Kensho Technologies as a software infrastructure engineer. Without having the full computer science experience that my CS major peers had, it was a big leap into such a foundational field in the tech industry. And as with most first jobs out of college, it came with a big learning curve. In fact, I think the curve is more like a mountain range with some peaks just a little too far to see until you reach the peak of the current concept/tool/problem you’re facing. It’s like you think you’ve learned so much, and then BAM! you realize that you’ve opened the door to so much more. (Spoiler alert: I like that part of the job the most!) Now, what a software infrastructure engineer is and does will vary from company to company, but this is what my experience is and how I see the role after 1.5 years at Kensho!
Being the foundation
It’s kind of obvious from the title, but software infrastructure engineers maintain the infrastructure for the software of a company. The way I see it, we’re the software engineers who create, manage, and maintain the tools and services that the other software engineers use. This could involve creating internal tools for company-specific problems, managing and monitoring cloud deployments, or alerting other teams of changes that could affect their work. For example, at Kensho I have worked on upgrading parts of our monitoring services (specifically logging and tracing), supplementing our policy engine (an internal continuous integration tool that checks for standardization across our codebase), and debugging errors with the Python packaging system we use. As a result of the variety of projects, we have the benefit of becoming knowledgeable in a variety of areas. Though, sometimes I do resonate with Albert Einstein when he said, “The more I learn, the more I realize how much I don’t know”.
Your new best friends
Going from small group projects to working with code updated by dozens of engineers, Kensho’s codebase is by far the biggest I’ve worked on, and because of its size, manually maintaining it like I used to do for simple projects in college is just not an option. That’s where continuous integration and continuous delivery/deployment (CI/CD) come in. The former is the process of iteratively integrating code that many engineers could simultaneously be working on into one singular codebase, while the latter refers to packaging, building, and testing that codebase to be hopefully deployed for use by end users. At Kensho, some of the tools we use are Github Enterprise and Jenkins. But even though this is great for projects handled by multiple teams, it’s easy to create your own CI/CD pipeline for personal projects using Github and Github Actions. I look back at a lot of the problems I faced in college when I or someone else pushed a commit that “broke the code” but could’ve easily been avoided with some automated tests. You live and you learn….and you test.
Containers in the sky
Before Kensho, the closest I had come to cloud computing and distributed systems was saving problem sets in Google Drive. Now, I use tools like Docker, Kubernetes, and AWS daily. I’ve realized that you can’t get far by being restricted to one environment. Learning about the benefits of containerizing code, how to create a cohesive containerized environment through container orchestration, and deploying code to cloud environments is at the heart of my job. And as infrastructure engineers, our job is to make sure that other teams can interact with these tools comfortably and sometimes in an automated way. For example, we create base images (templates used to create containers) that may contain certain dependencies and environment settings from which other teams can build their own images. This takes a lot of the tedious and repetitive work away from engineers who could spend more time working on their project.
With great power comes great responsibility
You may have guessed by now that in being so foundational, infrastructure engineers have a lot of power. Usually, it is up to us to hand out permissions to different services, work with the security team to ensure data isn’t compromised, and handle upgrades of tools and services for performance, security, and feature gains. As a result, the changes we introduce have consequences, which, as a new grad, did come with an ounce of fear with each commit. And although I wouldn’t say I’m always on my toes when I work, I think this role gives me a lot of perspective into my work’s impact. For instance, one time I was upgrading our Grafana instance and being confident that successful testing on our test cluster would migrate easily to our other clusters, I let the team know of expected downtime for the service and began the upgrade. But then I began running into trouble, and given that Grafana is used as a visualization tool for our services’ metrics, the prolonged downtime did have minor impacts on some teams’ workflows (nothing breaking thankfully). Eventually, the upgrade was successful, but my takeaway was how integral the work of infrastructure engineers was to the workflow of others, and it gave me a greater appreciation for my work.
Teamwork makes the dream work
As with most professions, in order to be successful at what you do, being able to work with others is key. There is constant dialogue between infra engineers and other teams on what’s working, what’s not working, whether a certain tool can be integrated into our environment, whether we can upgrade to the new version of a tool, etc. The list goes on and it’s exciting because the environment is dynamic, and you are at the forefront of its evolution. Working within Kensho’s friendly and collaborative culture is what I think keeps our productivity high, engagement strong, and products competitive. One of my favorite parts of my job (which ironically used to be the most anxious) is when I can answer people’s questions. It’s by far one of the most rewarding aspects as I can immediately see the outcome. This is not to say I always have the answer though! Being able to help someone debug or pointing them in the right direction is just as helpful.
1.5 years down, many to go
After only a year and a half in the role, I still have a lot to learn, but I’ve also learned more in that time than I could have imagined. As with any new role, the beginning can be rough and overwhelming, but hindsight is 20/20 as you come to appreciate newfound skills. Some of my key takeaways are:
It’s easy to go down a rabbit hole when learning a new concept or tool, so it can be beneficial to ask questions early from those who can provide a more concise explanation.
There is always room to contribute, so leaning into the challenges and offering what you can is valuable, and it allows you to enter spaces that promote your growth.
The environment in which you work is as important as what you work on. Because Kensho’s culture fosters exploration and collaboration, I feel accomplished as I work.
To sum it up, the more I learn, the more I can do, which makes me excited about what the next 1.5 years has in store for me.