Role at a glance
- Salary
- $181.1K – $245K/yr
- Location
- Seattle, Washington, United States
- Work arrangement
- On-site
- Employment
- Full-time
- Education
- Master's
Spotted an issue?
We’ll check it against the original posting.
Qualifications
Required
- 7+ years of working as a Technical Product Manager experience
- 5+ years of technical (software development, network development, IT, other related) experience
- Experience delivering large-scale SaaS, PaaS or LaaS products where you are responsible for the full product lifecycle, from concept through GTM (go to market)
- Experience with machine learning or AI systems, including familiarity with model evaluation, testing methodologies, or quality assurance frameworks
- Demonstrated ability to write clear, detailed technical product documents (PR/FAQs, technical specifications, architecture diagrams) and drive engineering teams to consensus
Preferred
- MBA or advanced degree in a technical discipline
- Experience with LLM evaluation, AI safety testing, or automated quality assurance for generative AI systems
- Track record of building and launching evaluation or testing platforms at scale
- Experience working with applied scientists to productize research into customer facing features
- Familiarity with AWS services (AgentCore, SageMaker) and agentic AI frameworks (LangFuse)
- Strong track record of shaping business strategy for technical products or services for large enterprises or partners
- Prior experience bringing to life 0 to 1 AI powered offerings
About the role
Original posting provided by amazon
As part of the AWS Applied AI Solutions organization, we have a vision to provide business applications, leveraging Amazon's unique experience and expertise, that are used by millions of companies worldwide to manage day-to-day operations. We will accomplish this by accelerating our customers' businesses through delivery of intuitive and differentiated technology solutions that solve enduring business challenges. We blend vision with curiosity and Amazon's real-world experience to build opinionated, turnkey solutions.
We are looking for a Principal Product Manager, Technical to own and drive the product strategy, revenue, and adoption for AI Agent Evaluations within our Core Services AI Foundations team. You will be responsible for defining the long-term vision and building the business around evaluation capabilities that enable development teams to measure, benchmark, and continuously improve the quality, safety, and reliability of their AI powered agents. This includes owning the full product lifecycle from concept through GTM for evaluation tooling, quality scoring methodologies, regression testing frameworks, and human review workflows. You will own one or more products in production while also getting in on the ground floor of new projects that bring evaluation capabilities to customers worldwide.
As a Principal Product Manager, you will be part of the larger product leadership community at AWS. This community plays a critical role in broad business planning, working closely with senior management to develop business targets and resource requirements, influencing long term technical and business strategy, helping hire a talented team of PMs, and enabling us to deliver solutions rapidly. You will be seen as the subject matter expert for AI evaluation within Amazon AI.
A successful candidate will bring a passion for technology services, strong business acumen and judgment, proven experience launching products and building business around them, ability to define products, desire to have an industry wide impact, and ability to work within a fast moving environment in a large company to rapidly deliver services that have broad business impact. You will work backward from the needs of development teams building agentic AI applications and define how they assess agent behavior across dimensions including correctness, safety, groundedness, and customer satisfaction. You will partner closely with engineering and applied science teams to translate complex evaluation methodologies into products that scale across diverse use cases and agent architectures.
Key job responsibilities
• Lead product definition for AI agent evaluation capabilities, owning the customer working backwards strategy, tenets, long term goals, and working backwards documents including customer and market feedback, competitive analysis, and business metrics to inform direction
• Define product vision including all aspects of future roadmap, investment, innovation, and experimentation for automated evaluation pipelines, human review workflows, and quality benchmarking tools
• Own execution of product planning and development including customer goals and business requirements for product releases, ensuring implementation is aligned with product goals, and ownership of product positioning
• Lead product launches owning GTM plans that deliver results ensuring customer and business goals are met in operational launch plans
• Create and revise GTM strategy and be responsible for pricing new features and updating pricing for existing features working closely with finance
• Work with field and BD teams to create customer funnel and drive POC to pilot to production motions
• Be responsible for creating and reporting on the service's revenue and customer traction through weekly and monthly business reports shared with leadership
• Partner with applied scientists and engineers to translate research into production grade product features
• Drive alignment across dependent teams (identity, observability, data platform) to ensure evaluation signals integrate seamlessly with the broader Core Services AI Foundations platform
• Lead interaction with the technical team including helping make tradeoffs based on customer requirements and QA/testing of the product
A day in the life
You split your time between building the business (developing partnership strategy, earning trust with stakeholders, defining GTM) and building the product (writing requirements, working closely with engineering, driving launches). On any given day you might be meeting with a customer's engineering leadership to understand how they assess agent quality at scale, presenting product strategy and business results to senior leadership, reviewing a pricing model with finance, working with field teams on a customer POC, or collaborating with applied scientists on how to productize new evaluation research. You operate with high independence across ambiguous problem spaces, make tradeoffs across competing priorities, and own outcomes from requirements through launch and sustained business growth.
About the team
ABOUT AWS:
Diverse Experiences
Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.
Why AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.
Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.
Inclusive Team Culture
Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity and AmazeCon conferences, inspire us to never stop embracing our uniqueness.
Mentorship and Career Growth
We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
- 5+ years of technical (software development, network development, IT, other related) experience
- Experience delivering large-scale SaaS, PaaS or LaaS products where you are responsible for the full product lifecycle, from concept through GTM (go to market)
- Experience with machine learning or AI systems, including familiarity with model evaluation, testing methodologies, or quality assurance frameworks
- Demonstrated ability to write clear, detailed technical product documents (PR/FAQs, technical specifications, architecture diagrams) and drive engineering teams to consensus
- Experience with LLM evaluation, AI safety testing, or automated quality assurance for generative AI systems
- Track record of building and launching evaluation or testing platforms at scale
- Experience working with applied scientists to productize research into customer facing features
- Familiarity with AWS services (AgentCore, SageMaker) and agentic AI frameworks (LangFuse)
- Strong track record of shaping business strategy for technical products or services for large enterprises or partners
- Prior experience bringing to life 0 to 1 AI powered offerings
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
USA, WA, Seattle - 181,100.00 - 245,000.00 USD annually
We are looking for a Principal Product Manager, Technical to own and drive the product strategy, revenue, and adoption for AI Agent Evaluations within our Core Services AI Foundations team. You will be responsible for defining the long-term vision and building the business around evaluation capabilities that enable development teams to measure, benchmark, and continuously improve the quality, safety, and reliability of their AI powered agents. This includes owning the full product lifecycle from concept through GTM for evaluation tooling, quality scoring methodologies, regression testing frameworks, and human review workflows. You will own one or more products in production while also getting in on the ground floor of new projects that bring evaluation capabilities to customers worldwide.
As a Principal Product Manager, you will be part of the larger product leadership community at AWS. This community plays a critical role in broad business planning, working closely with senior management to develop business targets and resource requirements, influencing long term technical and business strategy, helping hire a talented team of PMs, and enabling us to deliver solutions rapidly. You will be seen as the subject matter expert for AI evaluation within Amazon AI.
A successful candidate will bring a passion for technology services, strong business acumen and judgment, proven experience launching products and building business around them, ability to define products, desire to have an industry wide impact, and ability to work within a fast moving environment in a large company to rapidly deliver services that have broad business impact. You will work backward from the needs of development teams building agentic AI applications and define how they assess agent behavior across dimensions including correctness, safety, groundedness, and customer satisfaction. You will partner closely with engineering and applied science teams to translate complex evaluation methodologies into products that scale across diverse use cases and agent architectures.
Key job responsibilities
• Lead product definition for AI agent evaluation capabilities, owning the customer working backwards strategy, tenets, long term goals, and working backwards documents including customer and market feedback, competitive analysis, and business metrics to inform direction
• Define product vision including all aspects of future roadmap, investment, innovation, and experimentation for automated evaluation pipelines, human review workflows, and quality benchmarking tools
• Own execution of product planning and development including customer goals and business requirements for product releases, ensuring implementation is aligned with product goals, and ownership of product positioning
• Lead product launches owning GTM plans that deliver results ensuring customer and business goals are met in operational launch plans
• Create and revise GTM strategy and be responsible for pricing new features and updating pricing for existing features working closely with finance
• Work with field and BD teams to create customer funnel and drive POC to pilot to production motions
• Be responsible for creating and reporting on the service's revenue and customer traction through weekly and monthly business reports shared with leadership
• Partner with applied scientists and engineers to translate research into production grade product features
• Drive alignment across dependent teams (identity, observability, data platform) to ensure evaluation signals integrate seamlessly with the broader Core Services AI Foundations platform
• Lead interaction with the technical team including helping make tradeoffs based on customer requirements and QA/testing of the product
A day in the life
You split your time between building the business (developing partnership strategy, earning trust with stakeholders, defining GTM) and building the product (writing requirements, working closely with engineering, driving launches). On any given day you might be meeting with a customer's engineering leadership to understand how they assess agent quality at scale, presenting product strategy and business results to senior leadership, reviewing a pricing model with finance, working with field teams on a customer POC, or collaborating with applied scientists on how to productize new evaluation research. You operate with high independence across ambiguous problem spaces, make tradeoffs across competing priorities, and own outcomes from requirements through launch and sustained business growth.
About the team
ABOUT AWS:
Diverse Experiences
Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.
Why AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.
Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.
Inclusive Team Culture
Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity and AmazeCon conferences, inspire us to never stop embracing our uniqueness.
Mentorship and Career Growth
We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
Basic Qualifications
- 7+ years of working as a Technical Product Manager experience- 5+ years of technical (software development, network development, IT, other related) experience
- Experience delivering large-scale SaaS, PaaS or LaaS products where you are responsible for the full product lifecycle, from concept through GTM (go to market)
- Experience with machine learning or AI systems, including familiarity with model evaluation, testing methodologies, or quality assurance frameworks
- Demonstrated ability to write clear, detailed technical product documents (PR/FAQs, technical specifications, architecture diagrams) and drive engineering teams to consensus
Preferred Qualifications
- MBA or advanced degree in a technical discipline- Experience with LLM evaluation, AI safety testing, or automated quality assurance for generative AI systems
- Track record of building and launching evaluation or testing platforms at scale
- Experience working with applied scientists to productize research into customer facing features
- Familiarity with AWS services (AgentCore, SageMaker) and agentic AI frameworks (LangFuse)
- Strong track record of shaping business strategy for technical products or services for large enterprises or partners
- Prior experience bringing to life 0 to 1 AI powered offerings
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, WA, Seattle - 181,100.00 - 245,000.00 USD annually
About the company
amazon
Large Enterprise
Amazon is a global leader in e-commerce and cloud computing, founded in 1994 by Jeff Bezos. Initially starting as an online bookstore, it has since expanded its offerings to include a vast range of products and services, including electronics, fashion, and digital content. With Amazon Web Services (AWS), the company also provides powerful cloud solutions to businesses around the world. Known for its innovation, customer-centric approach, and commitment to operational efficiency, Amazon continues to shape the future of retail and technology, consistently seeking new ways to enhance customer experiences.
Amazon is a global leader in e-commerce and cloud computing, founded in 1994 by Jeff Bezos. Initially starting as an online bookstore, it has since expanded its offerings to include a vast range of products and services, including electronics, fashion, and digital content. With Amazon Web Services (AWS), the company also provides powerful cloud solutions to businesses around the world. Known for its innovation, customer-centric approach, and commitment to operational efficiency, Amazon continues to shape the future of retail and technology, consistently seeking new ways to enhance customer experiences.