Skip to main content

Hiring SREs

Hiring engineers is a challenging task. Doing it right can be difficult, and many companies struggle with it. I gave a talk at the Velocity conference a while back discussing how LinkedIn’s SRE team solved this problem and how I designed a hiring process that’s fair, interesting, and gets results. If you don’t want to read the whole post, you can just watch the video below.
Before talking about the actual mechanics of the interview process, though, I want to talk a little bit about the theory behind interviewing and hiring. In any circumstance where you’re doing hiring, you have a specific need that you need folks to fill. There’s a task that needs to be done, a role that needs to be occupied, and business metrics that need to be moved. Before you can really start doing any hiring, you have to know definitively what those are. You also need to know what skills are the minimum necessary to meet your requirements. Most importantly, you need to make sure that you’re only testing for those skills.
Testing for other stuff might make you feel good, or make your team think that the hiring process is really hard, so they must be very smart, but it will be incredibly damaging in the long run to your goals of getting the right people doing the right things. It’s critical that before you start rushing out to redesign your hiring process, you know—and make sure everyone internally is aligned with—what people need to know to do the job.
Once you’ve done that, the rest of the hiring process should center around two things: efficiency and testing the ability to do the job. From an efficiency perspective, developing screens that can be given over the phone or online can help you scale. From the “ability to do the job” perspective, developing screens that test the essential skills (and only the essential skills) makes sure that your selection process gets you people who can do the job, but who also view your selection process as fair and maybe even fun.
The process at LinkedIn begins with a quick phone screen given by the recruiter. The recruiter discusses the candidate’s availability and gives general information about the role and LinkedIn. He or she then asks seven questions with very simple answers, intended primarily to make sure that the candidate has at least some elementary knowledge in everything that is needed for the role. These questions were written to have as unambiguous an answer as possible, and to not require any technical knowledge on the part of the interviewer. One example question from our recruiter phone screen is “For each IP address I give you, tell me whether it is publicly-routable on the global internet or not.”
Once a candidate passes the recruiter screen, we’re confident that a conversation with an engineer has at least a good chance of being productive, so we schedule a coding-focused phone screen. We do this because the SRE role at LinkedIn involves both systems knowledge and development knowledge, so we need to make sure our candidates can produce code. We use Coderpad, but any collaborative typing system works for doing this. In the coding phone screen, we use a progressive method to ease the candidate into the right frame of mind for doing the coding, and to also let us bail out on the interview with some degree of grace if it’s clear the candidate can’t code. We start with a very elementary problem (run through a loop and print some things) and end with a task like one our SREs could be expected to do on any given day (connect to a RESTful API and retrieve some data, then process it.)
Once we’re confident the candidate can code enough to be successful, we advance the candidate to a more in-depth systems and architecture phone screen. Part of the reason for conducting so many phone screens instead of just bringing the candidates onsite is that an onsite is very expensive—both for us and for the candidate—so we want to feel confident that candidates we invite onsite will pass. The operations phone screen involves three questions, one about how systems work, one about performing a task at large scale, and a third about monitoring and alerting.
Once the candidate has cleared both engineering phone screens, we invite them to campus for a day. Possibly the most important piece of advice here is to try very hard to avoid repeating any of the activities you’ve already had the candidate do. In our case, we’ve had candidates write code, and we’ve had candidates discuss some low-level details about how UNIX-hosted network services work at scale. So, for our onsite interview, we ask them to do different things.
The signature task we have candidates for SRE perform in the onsite is an exercise called “Live Troubleshooting,” which is exactly what it sounds like: we give the candidate a laptop running a service that is broken in various ways and ask the candidate to fix it. It’s about as real a simulation as you can get for what a night on-call is like as an SRE.
SREs are also responsible for identifying the cause of incidents, so we have candidates go through a module where we ask them to do that. A sample alert dashboard is given to the candidate, and they have to triage the alerts and write status reports for their teammates and management about what went wrong and what they’re fixing based on the alerts they see.
Additionally, SREs must be able to produce code, but probably more importantly, they’re responsible for being able to read and understand code. To test this ability, we give our candidates a code review exercise, consisting of a few Pythonish pieces of code with several errors or poor design choices, and ask the candidates to conduct a code review on them, like they would with their co-workers.
One of the biggest benefits of our bar for SRE talent being so high is that SREs have a high level of trust and involvement in large-scale decisions, like how to architect a new service. Services at LinkedIn can handle millions of queries per second, so understanding how to architect a scalable, reliable service is key. We ask our candidates to design a service from the ground up to accomplish a simple task (like a link shortener or an image host.)
Finally, we need to make sure that candidates understand what it’s like being an SRE and how they work with others. To do this, we have the candidates speak with an SRE manager and have lunch with at least one other SRE. These opportunities let us learn more about the candidate in a less-structured environment and also give the candidate a chance to ask more questions about the role and the company.
You’ll notice what’s missing in our hiring process: trivia, whiteboard coding, in-depth kernel knowledge as an expectation, and other hoop-jumping exercises that don’t make people better SREs. We worked very hard to determine what the most essential skills are for the SRE role as it stands at LinkedIn, and structured our hiring process to give us the most possible information about a candidate’s likely performance in that role.
The lesson learned here is this: if your hiring process reflects the actual job, then you can tell a lot about how good a fit someone is for the role based on their reaction to the hiring process. If they hate being asked to do simulations of what they’ll actually do on the job, the candidate may realize that the job is a bad fit for them off the bat, saving you from a bad hire. Candidates have given us so much positive feedback about the process, including many who describe it as very hard, but fun. This is exactly the reaction we were hoping for.

Comments

Popular posts from this blog

OWASP Top 10 Threats and Mitigations Exam - Single Select

Last updated 4 Aug 11 Course Title: OWASP Top 10 Threats and Mitigation Exam Questions - Single Select 1) Which of the following consequences is most likely to occur due to an injection attack? Spoofing Cross-site request forgery Denial of service   Correct Insecure direct object references 2) Your application is created using a language that does not support a clear distinction between code and data. Which vulnerability is most likely to occur in your application? Injection   Correct Insecure direct object references Failure to restrict URL access Insufficient transport layer protection 3) Which of the following scenarios is most likely to cause an injection attack? Unvalidated input is embedded in an instruction stream.   Correct Unvalidated input can be distinguished from valid instructions. A Web application does not validate a client’s access to a resource. A Web action performs an operation on behalf of the user without checking a shared sec

CKA Simulator Kubernetes 1.22

  https://killer.sh Pre Setup Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these: alias k = kubectl                         # will already be pre-configured export do = "--dry-run=client -o yaml"     # k get pod x $do export now = "--force --grace-period 0"   # k delete pod x $now Vim To make vim use 2 spaces for a tab edit ~/.vimrc to contain: set tabstop=2 set expandtab set shiftwidth=2 More setup suggestions are in the tips section .     Question 1 | Contexts Task weight: 1%   You have access to multiple clusters from your main terminal through kubectl contexts. Write all those context names into /opt/course/1/contexts . Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh , the command should use kubectl . Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh , but without the use of k

标 题: 关于Daniel Guo 律师

发信人: q123452017 (水天一色), 信区: I140 标  题: 关于Daniel Guo 律师 关键字: Daniel Guo 发信站: BBS 未名空间站 (Thu Apr 26 02:11:35 2018, 美东) 这些是lz根据亲身经历在 Immigration版上发的帖以及一些关于Daniel Guo 律师的回 帖,希望大家不要被一些马甲帖广告帖所骗,慎重考虑选择律师。 WG 和Guo两家律师对比 1. fully refund的合约上的区别 wegreened家是case不过只要第二次没有file就可以fully refund。郭家是要两次case 没过才给refund,而且只要第二次pl draft好律师就可以不退任何律师费。 2. 回信速度 wegreened家一般24小时内回信。郭律师是在可以快速回复的时候才回复很快,对于需 要时间回复或者是不愿意给出确切答复的时候就回复的比较慢。 比如:lz问过郭律师他们律所在nsc区域最近eb1a的通过率,大家也知道nsc现在杀手如 云,但是郭律师过了两天只回复说让秘书update最近的case然后去网页上查,但是上面 并没有写明tsc还是nsc。 lz还问过郭律师关于准备ps (他要求的文件)的一些问题,模版上有的东西不是很清 楚,但是他一般就是把模版上的东西再copy一遍发过来。 3. 材料区别 (推荐信) 因为我只收到郭律师写的推荐信,所以可以比下两家推荐信 wegreened家推荐信写的比较长,而且每封推荐信会用不同的语气和风格,会包含lz写 的research summary里面的某个方面 郭家四封推荐信都是一个格式,一种语气,连地址,信的称呼都是一样的,怎么看四封 推荐信都是同一个人写出来的。套路基本都是第一段目的,第二段介绍推荐人,第三段 某篇或几篇文章的abstract,最后结论 4. 前期材料准备 wegreened家要按照他们的模版准备一个十几页的research summary。 郭律师在签约之前说的是只需要准备五页左右的summary,但是在lz签完约收到推荐信 ,郭律师又发来一个很长的ps要lz自己填,而且和pl的格式基本差不多。 总结下来,申请自己上心最重要。但是如果选律师,lz更倾向于wegreened,