快手大数据调度&大模型训练AI-Infra工程师/专家
社招全职3-5年J0012地点:北京状态:招聘
任职要求
1、有大数据引擎(Spark&Flink)、大模型训练AI-Infra优化经验优先,参与过大规模分布式系统开源贡献,或在相关领域有论文产出者加分; 2、熟悉K8S&了解Yarn调度,熟悉机器学习相关技术,有规范的文档撰写习惯,关注业界计算基础…
登录查看完整任职要求
微信扫码,1秒登录
工作职责
1、负责大数据方向Flink&Spark on K8S的关键能力建设,包括:Spark 高吞吐调度、Flink 极致负载均衡、Spark 计算 Offload 至 GPU 的调度支持、大数据作业资源弹性调度等; 2、负责大模型训练的ETTR(端到端训练启动时间)优化与 MFU(模型计算利用率)提升的关键技术能力建设,包括:GPU容器冷/热快速启停技术、模型服务初始化阶段的数据预加载与加速技术等。
包括英文材料
大数据+
https://www.youtube.com/watch?v=bAyrObl7TYE
https://www.youtube.com/watch?v=H4bf_uuMC-g
With all this talk of Big Data, we got Rebecca Tickle to explain just what makes data into Big Data.
Spark+
[英文] Learning Spark Book
https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf
This new edition has been updated to reflect Apache Spark’s evolution through Spark 2.x and Spark 3.0, including its expanded ecosystem of built-in and external data sources, machine learning, and streaming technologies with which Spark is tightly integrated.
Flink+
https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/learn-flink/overview/
This training presents an introduction to Apache Flink that includes just enough to get you started writing scalable streaming ETL, analytics, and event-driven applications, while leaving out a lot of (ultimately important) details.
https://www.youtube.com/watch?v=WajYe9iA2Uk&list=PLa7VYi0yPIH2GTo3vRtX8w9tgNTTyYSux
Today’s businesses are increasingly software-defined, and their business processes are being automated. Whether it’s orders and shipments, or downloads and clicks, business events can always be streamed. Flink can be used to manipulate, process, and react to these streaming events as they occur.
大模型+
https://www.youtube.com/watch?v=xZDB1naRUlk
You will build projects with LLMs that will enable you to create dynamic interfaces, interact with vast amounts of text data, and even empower LLMs with the capability to browse the internet for research papers.
https://www.youtube.com/watch?v=zjkBMFhNj_g
分布式系统+
https://www.distributedsystemscourse.com/
The home page of a free online class in distributed systems.
https://www.youtube.com/watch?v=7VbL89mKK3M&list=PLOE1GTZ5ouRPbpTnrZ3Wqjamfwn_Q5Y9A
Kubernetes+
https://kubernetes.io/docs/tutorials/kubernetes-basics/
This tutorial provides a walkthrough of the basics of the Kubernetes cluster orchestration system.
https://kubernetes.io/zh-cn/docs/tutorials/kubernetes-basics/
本教程介绍 Kubernetes 集群编排系统的基础知识。每个模块包含关于 Kubernetes 主要特性和概念的一些背景信息,还包括一个在线教程供你学习。
https://www.youtube.com/watch?v=s_o8dwzRlu4
Hands-On Kubernetes Tutorial | Learn Kubernetes in 1 Hour - Kubernetes Course for Beginners
https://www.youtube.com/watch?v=X48VuDVv0do
Full Kubernetes Tutorial | Kubernetes Course | Hands-on course with a lot of demos
还有更多 •••
相关职位
社招3年以上大数据(数据计算
1.设计并实现高性能、高可用的分布式任务调度引擎,支持分钟级百万任务调度; 2.构建分布式调度系统的高可用架构,实现故障自动转移与恢复; 3.设计并实现监控告警体系,推动调度系统的可观测性建设; 4.推动代码质量、单元测试和工程规范落地。
更新于 2026-07-13深圳
社招技术类
1. 负责⼤数据资源调度系统YARN 的设计和研发⼯作,解决集群规模增⻓带来的技术挑战,提⾼集群稳定性、可扩展性,深度优化资源调度策略和性能,提升资源利⽤效率; 2. 负责 公司在离线混部系统 的设计和研发⼯作,解决混部场景下,在离线资源复⽤、协调、隔离等问题,保证业务稳定性,提升在离线集群的资源利⽤效率; 3. 深⼊理解系统软硬件特性,关注线上运⾏状态,分析和解决线上问题,推动业务需求的解决⽅案落地。
更新于 2026-06-16上海
社招技术类
1. 负责⼤数据资源调度系统YARN 的设计和研发⼯作,解决集群规模增⻓带来的技术挑战,提⾼集群稳定性、 可扩展性,深度优化资源调度策略和性能,提升资源利⽤效率。 2. 负责 公司在离线混部系统 的设计和研发⼯作,解决混部场景下,在离线资源复⽤、协调、隔离等问题,保证 业务稳定性,提升在离线集群的资源利⽤效率。 3. 深⼊理解系统软硬件特性,关注线上运⾏状态,分析和解决线上问题,推动业务需求的解决⽅案落地
更新于 2026-06-18上海