Sr. Software Engineer, Observability

The Monitoring Infrastructure team in the Observability & Performance group at Slack develops platforms and tools which produce telemetry, provide insights, and improves observability in Slack production services with a focus on performance and reliability. We develop management tools for distributed applications and infrastructure, maintain datasets for performance and system analytics, and build interfaces and backend systems to answer questions and infer behavioral patterns about our users and systems. Our toolset is varied,  we work with open-source observability/monitoring technologies like Elasticsearch and Prometheus, or cloud providers such as AWS, and develop software using a combination of Go, Python, or Java.

As part of the Monitoring Infrastructure team, you will be focused on our log pipelines and work closely with other teams in engineering, product development, and customer experience to provide valuable insights to drive decisions and ensure a positive user experience for our Slack customers. You will also help build and maintain distributed services in an environment that processes millions of data points per second with the ability to self-heal and scale up or down to meet demand. We are an inclusive team with deep empathy for our colleagues and customers.

You can see the team at work here at Monitorama 2018.

Slack has a positive, diverse, and supportive culture—we look for people who are curious, inventive, and work to be a little better every single day. In our work together we aim to be smart, humble, hardworking, and, above all, collaborative. If this sounds like a good fit for you, why not say hello?

About the Role

This is a remote senior engineering position based in the United States or Canada.

What you will be doing

  • Build, maintain, and ensure timely delivery of our high-volume event log pipelines.
  • Create libraries, tools, and automation to help ensure that critical event data gets to the right place.
  • Encouraging a culture of Observability at Slack - help suss out problem areas and consult on improving visibility into our systems.
  • Prototyping tooling interfaces or building new features for engineering use cases.
  • Improving auto-remediation in our logging infrastructure to avoid recurring failures.
  • Teaching engineers or customer experience agents how to use our tools to introspect their systems.
  • Participating in the Monitoring Infrastructure on-call rotation, triaging, and addressing production issues as they arise.

What you should have

  • You are a strong communicator. Explaining complex technical concepts to designers, support, and other engineers is no problem for you.
  • You enjoy helping onboard new team members, mentoring, and teaching others.
  • You live for unit tests, code review, design documentation, debugging, and solving problems.
  • You have a deep curiosity about how things work under the hood.
  • You are motivated by helping others succeed. When things break — and they will — you are eager and able to help fix things. You like thinking of ways to improve efficiency or bring delight to your coworkers.
  • You care about improving the performance of systems through data-informed decisions.
  • You also know that the internet is a scary place and understand security concepts deeply and can put them into action to protect us and our users.



  • Firm grasp of computer science fundamentals: data structures, algorithms, programming languages, distributed systems, and information retrieval.
  • Bachelor's degree in Computer Science, Engineering or related field, or equivalent training, fellowship, or work experience


  • Experience with functional or imperative programming languages -- e.g., PHP, Python, Go, C, or Java (used without frameworks).
  • Experience with creating interfaces, tooling, or automation to help define a path for engineers to self-service.
  • Experience deploying, operating, and debugging server software on Linux at scale

Bonus Points:

  • Passionate about data visualization, graphing, and maximizing signal versus noise.
  • Experience with Elasticsearch, Logstash, and Kibana.
  • Solid competency with Prometheus, OpenTracing, or any other widely-used monitoring/visibility platform.
  • Prior experience with or knowledge of large scale, high volume distributed systems, distributed databases, and data pipelines.
  • Experience with containerization frameworks such as Kubernetes.
  • Experience using deployment automation/configuration management, especially Terraform or Chef.
  • Experience with AWS and other virtualized environments.
  • Experience with message queue services, such as Kafka.

For Colorado-based roles: Minimum annual salary of $142,000. This base pay is for illustrative purposes only. This position is eligible for additional compensation and benefits including: incentive compensation; health benefits; flexible spending account; retirement benefits; life insurance; commuter benefits; paid time off (including PTO, emergency time off, paid sick leave, medical leave, volunteer time off, civic duty leave, bereavement leave, floating holidays and paid holidays); parental leave and benefits; mobile phone and internet allowance; perks stipend; and other employee perks and benefits. 

The actual offer, reflecting the total compensation package and benefits, will be at the company’s sole discretion, and determined by a myriad of factors including, but not limited to, years of experience, depth of experience, and other relevant business considerations. The company also reserves the right to amend or modify employee perks and benefits at any time.

At Salesforce we believe that the business of business is to improve the state of our world. Each of us has a responsibility to drive Equality in our communities and workplaces. We are committed to creating a workforce that reflects society through inclusive programs and initiatives such as equal pay, employee resource groups, inclusive benefits, and more. Learn more about Equality at Salesforce and explore our benefits. and are Equal Employment Opportunity and Affirmative Action Employers. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status. and will not pay any third-party agency or company that does not have a signed agreement with or

Salesforce welcomes all.