Title: Site Reliability Engineer

Location: Reston, VA; Seattle, WA; or Austin, TX

Clearance Level: TS/SCI w/ Polygraph

Job Description:

Are you someone with a passion for taking on big challenges? Are you interested in cross-domain networks and large-scale, highly distributed systems? If so, then Oracle’s Cloud Infrastructure team is for you. We’re looking for Site Reliability Engineers (SRE’s) to build and support highly distributed systems, platform services, and tools for a highly distributed multi-tenant cloud environment at massive scale.

The Oracle Cloud Infrastructure (OCI) team can provide you the opportunity to build and operate a suite of massive scale, integrated cloud services in a broadly distributed, multi-tenant cloud environment. OCI is committed to providing the best in cloud products that meet the needs of customers who are tackling some of the world’s biggest challenges. This is a key role within the Oracle National Security Regions (ONSR) SRE team, with a strong emphasis in building solutions that support the delivery of incident and problem management.

We are looking for strong SRE’s to aid us in engineering and supporting cross-domain service solutions for scale and performance while providing strong guarantees of availability to our customers. Your team will have diverse expertise in systems, networking, and software development to provide the stability, performance, and reliability our customers need. You will build tools to address hard operational problems while enabling our service developers to remain agile and deliver feature changes. You will deliver the solutions that directly contribute to our customer’s success.

SREs will spend a significant amount of the role doing Tier-3 level Ops-related work, such as troubleshooting production issues and providing on-call support. When not working on operations the SRE is working on software engineering tasks such as optimizing our implementation of cross domain network solutions, and developing tools/systems that increase reliability, scalability and reduce operational overhead through automation. The ideal SRE candidate is proficient programmer who has large breadth of knowledge and experience including areas such as networking, internet protocols and Linux systems.

Our SREs are close partners with service owners in all aspects of service operations and ownership. We have a strong culture of innovation, collaboration and team work inspired by the DevOps philosophy.

Responsibilities:

• Provide Operations and Engineering support over a cross-domain solution in a restricted cloud environment.
• Build systems to address hard operational problems such as automation, security, scaling, availability, and resiliency - while enabling our service developers to remain agile and deliver feature changes.
• Value simplicity and usability as well as security and work comfortably in a collaborative, agile environment.
• Practice sustainable incident response and drive root case analysis.
• Take an active role in the definition and evolution of standard practices and procedures for our team and org.
• Deliver solutions that directly contribute to our customer’s success.
• Work closely with development team on maintaining operational health of core services for availability and low latency.
• Managing and triaging tickets. Driving prioritization and execution of work based on impact.
• Drive new runbooks to help reduce mean triage time of incidents. Prioritize and automate high hit count runbooks as appropriate.

Minimum Qualifications:

• Ability to maintain a TS/SCI w/ Polygraph clearance.

• Proficient with writing services/task automation in Python, Bash, Ruby, Perl, JavaScript, or Java
• Proficient with communication skills (writing, organization, learning exchange)
• Familiarity with core protocols (DNS, DHCP, HTTP, TCP)
• Deep knowledge of Linux internals and host-based networking
• Expert Linux/Unix performance and stability troubleshooting skills
• Familiarity with configuration management solutions such as Chef, Puppet, etc.
• Experience with devising, managing, and extending monitoring solutions for large scale environments.
• Systematic problem-solving approach, strong communication skills, a sense of ownership and drive
• Deep understand of service metrics and alarms through the development of dashboards, service KPIs, alarming systems
• Experience working in an operational environment with mission critical tier one services with associated pager duty
• 5+ years managing large scale, highly distributed, services infrastructures
• 2+ years managing host virtualization technologies (KVM, Containers, Docker, etc.)

Preferred Qualifications:

• Familiarity with cross-domain network solutions
• Proficient in coding complex, distributed systems using Python, Ruby, Java, or C/C++
• Deep knowledge of Networking (TCP, UDP, DNS, DHCP, IPSec)
• Deep focus on building secure Internet facing systems and services in hostile environments
• Experience in database management (Oracle DB, MYSQL, Postgres)
• Experience in shared file systems (Gluster, ZFS, etc.)
• 3+ years of experience in production software development with Agile methodologies
• 3+ years managing host, network, or storage virtualization technologies
• Expert troubleshooting skills
• Expert fleet automation and management solutions

Additional Responsibilities:

Work with Site Reliability Engineering (SRE) team on the shared full stack ownership of a collection of services and/or technology areas, with a particular focus in cross-domain network solutions. Understand the end-to-end configuration, technical dependencies, and overall behavioral characteristics of production services. Responsible for the design and delivery of the mission critical stack, with focus on security, resiliency, scale, and performance. Authority for end-to-end performance and operability. Partner with development teams in defining and implementing improvements in service architecture. Articulate technical characteristics of services and technology areas and guide Development Teams to engineer and add premier capabilities to the Oracle Cloud service portfolio. Understand and communicate the scale, capacity, security, performance attributes, and requirements of the service and technology stack. Demonstrate clear understanding of automation and orchestration principles. Act as ultimate escalation point for complex or critical issues that have not yet been documented as Standard Operating Procedures (SOPs). Utilize a deep understanding of service topology and their dependencies required to troubleshoot issues and define mitigations. Understand and explain the affect of product architecture decisions on distributed systems. Professional curiosity and a desire to a develop deep understanding of services and technologies.