字节跳动Content Understanding Multimodal Model Algorithm Engineer | 内容理解多模态大模型算法工程师-电商-筋斗云人才计划

Team Introduction: TikTok is a global short-video platform available in 150 countries and regions. Our mission is to inspire creativity and bring joy by helping users discover real and interesting moments that make life better. TikTok's global headquarters are in Los Angeles and Singapore, and we also have offices in New York City, London, Dublin, Paris, Berlin, Dubai, Jakarta, Seoul, and Tokyo. TikTok Research & Development (R&D) Team: The TikTok R&D team is dedicated to building and maintaining industry-leading products that drive the success of TikTok’s global business. By joining us, you'll work on core scenarios such as user growth, social features, live streaming, e-commerce consumer side, content creation, and content consumption, helping our products scale rapidly across global markets. You'll also face deep technical challenges in areas like service architecture and infrastructure engineering, ensuring our systems operate with high quality, efficiency, and security. Meanwhile, our team also provides comprehensive technical solutions across diverse business needs, continuously optimizing product metrics and improving user experience. Here, you'll collaborate with leading experts in exploring cutting-edge technologies and pushing the boundaries of what's possible. Every line of your code will serve hundreds of millions of users. Our team is professional and goal-oriented, with an egalitarian and easy-going collaborative environment. Research Project Introduction: With the advancement of hardware computing and the continuous breakthroughs of large models in CV/NLP/multimodal learning and even AGI fields, the large computing driven in recommendation scenarios are increasingly capable of capturing user preferences in a more comprehensive and nuanced way. This enables a deeper understanding of user needs and the discovery of latent interests, ultimately leading to enhanced user experiences. As a critical component of short video recommendation systems, the ranking module is responsible for fine-grained matching between users and content, selecting the videos users are most likely to be engaged with. In this context, the key research focus is how to best leverage large computing to maximize the model’s memory, generalization, and reasoning capabilities. 团队介绍： TikTok是一个覆盖150个国家和地区的国际短视频平台，我们希望通过TikTok发现真实、有趣的瞬间，让生活更美好。TikTok 在全球各地设有办公室，全球总部位于洛杉矶和新加坡，办公地点还包括纽约、伦敦、都柏林、巴黎、柏林、迪拜、雅加达、首尔和东京等多个城市。 TikTok研发团队，旨在实现TikTok业务的研发工作，搭建及维护业界领先的产品。加入我们，你能接触到包括用户增长、社交、直播、电商C端、内容创造、内容消费等核心业务场景，支持产品在全球赛道上高速发展；也能接触到包括服务架构、基础技术等方向上的技术挑战，保障业务持续高质量、高效率、且安全地为用户服务；同时还能为不同业务场景提供全面的技术解决方案，优化各项产品指标及用户体验。 在这里， 有大牛带队与大家一同不断探索前沿， 突破想象空间。 在这里，你的每一行代码都将服务亿万用户。在这里，团队专业且纯粹，合作氛围平等且轻松。 课题介绍： 随着硬件算力的发展以及大模型在CV/NLP/多模态以至于AGI领域的不断突破，推荐场景下的大算力驱动能够帮助模型更全面深刻理解用户偏好，进而更好地理解用户需求，挖掘用户潜在兴趣，进而带来更好地用户体验。排序模块作为整个短视频推荐系统中非常重要的一环，承载着用户与视频之间的细粒度匹配挖掘进而挑选出用户最感兴趣的视频。如何找到合适的路径来最大化大算力下模型的记忆、泛化、推理能力，成为了研究的重中之重。

Team Introduction: The TikTok AI Innovation Center is a department focused on building AI infrastructure and driving cutting-edge research in AI. We explore industry-leading AI technologies, including large language models (LLMs) and multimodal large models, with the goal of developing models that can understand multilingual content and vast amounts of video data, ultimately delivering a better content consumption experience for users. In the Code AI domain, we leverage the powerful code understanding and reasoning capabilities of LLMs to enhance program performance and R&D efficiency. Project Introduction: Multimodal foundation large models (VLM) represent a research hotspot in the industry and a critical technology for TikTok's business scenario applications. In 2024, TikTok's Innovation Center developed VFM V1, a multimodal large model tailored for TikTok's business scenarios. It matches the performance of the best open-source model Qwen VL on public test sets, while significantly outperforming all other foundation models on TikTok's business test sets. In the future, we aim to continuously develop foundation models with efficient perception and reasoning capabilities, capable of handling multilingual and massive video content understanding algorithms to deliver a better content consumption experience for users. Project Challenges: Enhance the multimodal perception encoder: The current encoder uses a fixed frame rate. We need to explore more efficient adaptive frame rates while considering the integration of modalities such as audio and user behavior. How to fuse multimodal perception and thinking capabilities to promote stronger comprehensive perception and cognitive abilities of the model. 团队介绍： TikTok AI创新中心，是致力于AI基础设施建设和创新研究的部门，探索行业领先的人工智能技术，包括大语言模型，多模态大模型等研究方向。我们希望研发能够处理多语言和海量视频内容理解的模型算法，为用户带来更好的内容消费体验。在Code AI方向，我们利用大语言模型强大的代码理解与推理能力，提升程序性能与研发效率。 课题介绍： 多模态基础大模型VLM 是行业的研究热点，也是TikTok业务场景应用的关键技术，2024年TikTok AI创新中心研发了面向TikTok业务场景的多模态大模型VFM V1，在公开测试集上能够与最好的开源模型 Qwen VL持平，同时在 TikTok 业务测试集上，能够大幅领先所有其它基础模型。未来，我们希望持续研发具有高效感知和推理思考能力的基础模型，能够处理多语言和海量视频内容理解的模型算法，为用户带来更好的内容消费体验。 课题挑战： 1、增强多模态感知编码器，当前的编码器是固定帧率，需要探索更高效的自适应帧率，同时考虑音频、用户行为等模态加入； 2、如何融合多模态感知和思考能力，促进更强的模型综合感知和认知能力。

Team Introduction: The Risk Control R&D Team is dedicated to addressing various challenges posed by malicious activities across ByteDance's products including Douyin and Toutiao. Their work spans multiple domains of risk governance such as content, transactions, traffic, and accounts. By leveraging technologies such as machine learning, multimodal models, and large models, the team strives to understand user behaviors and content, thereby identifying potential risks and issues. By continuously deepening their understanding of business and user behaviors, the team drives innovation in models and algorithms with an aim to build an industry-leading risk control algorithm system. Project Objectives: Optimize and enhance large models' ability to understand and reason about structured data (sequential data, graph data) based on risk control data. Project Necessity: Data in risk control scenarios is primarily structured, while large models have significantly improved their understanding of text and images. Integrating non-text/image structured data from risk control scenarios with large models to enable better comprehension of structured data remains an industry-wide challenge. This involves three key difficulties: 1. How to effectively align structured information with the NLP semantic space, allowing models to simultaneously understand both data structure and semantic information. 2. How to use appropriate instructions to enable large models to interpret structural information in structured data. 3. How to endow large language models with step-by-step reasoning capabilities for graph learning downstream tasks, thereby inferring more complex relationships and attributes. Project Content: Current industry explorations of structured data include: 1. Graph data understanding (e.g., GraphGPT: Enabling large models to read graph data, SIGIR'2024). 2. Graph data RAG (e.g., Microsoft GraphRAG: Unlocking LLM discovery on narrative private data). 3. Sequential data understanding (e.g., StructGPT: A large model reasoning framework for structured data, EMNLP-2023). However, current efforts mainly focus on understanding single-type structured data, and several challenges remain in risk control scenarios: 1. How to effectively fuse and understand various types of structured data, especially the integration of graph and sequential data. 2. Addressing the challenges mentioned in the ""Project Necessity"" section, particularly the step-by-step reasoning capabilities for downstream tasks, which are currently underexplored—especially reasoning over sequential data. Research Directions: 1. Large model structured data understanding 2. Large model structured data RAG 3. Large model thought chains 团队介绍： 风控研发团队致力于解决各个产品（包括抖音、头条等）面临的各种黑灰产对抗问题，涵盖内容、交易、流量、账号等多个方面的风险治理领域。利用机器学习、多模态、大模型等技术对用户行为、内容进行理解从而识别潜在的风险和问题。不断深入理解业务和用户行为，进行模型和算法创新，打造业界领先的风控算法体系。 课题介绍： 1、课题目标：以风控数据为基础，优化提高大模型对于结构化数据（序列数据、图数据）的理解推理能力； 2、课题背景：风控场景下的数据主要为结构化数据，而目前大模型对于文本和图像的理解能力有了很大的提升，如何跟风控场景的非文本、图像数据（结构化数据）结合起来，让大模型能够更好的理解结构化的数据，是一个业界难题。 面临着三大挑战 ： 1）如何有效地将结构化的信息与nlp语义空间进行对齐，使得模型能够同时理解数据结构和语义信息； 2）如何用适当的指令使得大模型理解结构化数据中的结构信息； 3）如何赋予大语言模型图学习下游任务的逐步推理能力，从而逐步推断出更复杂的关系和属性。 3、课题内容：目前业界对结构化数据探索有： 1）图数据理解相关GraphGPT：让大模型读懂图数据（SIGIR'2024）； 2）图数据RAG相关GraphRAG：Unlocking LLM discovery on narrative private data； 3）序列数据理解相关StructGPT：面向结构化数据的大模型推理框架（EMNLP-2023）。 目前的主要工作都是单一结构数据的理解，在风控场景下还面临几个问题： 1）对各种不同种类的的结构化数据融合理解怎么做，特别是融合图和序列数据的数据理解； 2）针对课题必要性中的问题； 3）对于下游任务的推理能力，目前的研究比较少，针对序列数据的推理能力研究非常少。 4、研究方向：大模型结构化数据理解、大模型结构化数据RAG、大模型思维链。

Team Introduction： The team primarily focuses on recommendation services for the International E-commerce Mall, covering information flow recommendation in core scenarios such as the mall homepage, transaction funnels, product detail pages, stores & showcases. Committed to providing hundreds of millions of users daily with precise and personalized recommendations for products, live streams, and short videos, the team dedicates itself to solving challenging problems in modern recommendation systems. Through algorithmic innovations, we continuously enhance user experience and efficiency, creating greater user and social value. Project Background/Objectives: This project aims to explore new paradigms for large models in the recommendation field, breaking through the long-standing structures of recommendation models and Infra solutions, achieving significantly better performance than current baseline models, and applying them across multiple business scenarios such as Douyin short videos/LIVE/E-commerce/Toutiao. Developing large models for recommendation is particularly challenging due to the high demands on engineering efficiency and the personalized nature of user recommendation experiences. The project will conduct in-depth research across the following directions to explore and establish large model solutions for recommendation scenarios: Project Challenges/Necessity: The emergence of LLMs in the natural language field has outperformed SOTA models in numerous vertical tasks. In contrast, industrial-grade recommendation systems have seen limited major innovations in recent years. This project seeks to revolutionize the long-standing paradigms of recommendation model architectures and Infra in the recommendation field, delivering models with significantly improved performance and applying them to scenarios like Douyin short video and LIVE. Key challenges include: High engineering efficiency requirements for recommendation systems; Personalized nature of user recommendation experiences; Effective content representation for media formats like short videos and live streams. The project will address these through deep research in model parameter scaling, content/user representation learning, multimodal content understanding, ultra-long sequence modeling, and generative recommendation models, driving systematic upgrades to recommendation models. Project Content: 1. Representation Learning Based on Content Understanding and User Behavior 2. Scaling of Recommendation Model Parameters and computing 3. Ultra-Long Sequence Modeling 4. Generative Recommendation Models Involved Research Directions: Recommendation Algorithms, Large Recommendation Models. 团队介绍： 推荐与营销团队，主要负责国际电商商城推荐业务，涵盖商城首页、交易链路、商品详情页、店铺&橱窗等多个核心场景的信息流推荐业务，致力于每天为亿量级用户提供精准个性化商品、直播、短视频推荐服务；团队致力于解决现代推荐系统中各种有挑战的问题，通过算法不断提升用户体验和效率、创造更大的用户和社会价值。 课题背景/目标： 本项目旨在探索推荐领域下的大模型新范式，突破现在持续了较长时间的推荐模型结构和Infra的方案，且效果大幅好于现在的基线模型，在抖音短视频/直播/电商/头条等多个业务场景上得到应用。推荐领域的大模型是比较有挑战的事情，推荐对工程效率的要求更高，且用户的推荐体验上是个性化的，本课题会以下多个方向来做深入的研究，探索和建设推荐场景的大模型方案。 课题挑战/必要性： 自然语言领域LLM的出现，效果在众多垂直任务上都好于sota模型，从推荐领域看过去工业级推荐系统在较长的时间没有大幅的变化过。本项目旨在探索推荐领域下的大模型方案，改变现在持续了较长时间的推荐模型结构和Infra的基本范式，且效果大幅好于现在的模型，在抖音短视频/直播等多个业务场景上得到应用。但是怎么做好推荐领域的大模型也是一个比较有挑战的事情，推荐对工程效率的要求更高，且用户的推荐体验上是个性化的，以及如何短视频、直播等体裁上做号内容的表征也是需要被解决的问题，这里会从模型参数scaling up、内容和用户的表征学习、内容理解多模态、超长序列建模、生成式推荐模型等多个方向来做深入的研究，对推荐场景的模型做系统性的升级。 课题内容： 1、基于内容理解和用户行为的表征学习； 2、推荐模型参数和算力scaling up； 3、超长序列建模； 4、生成式推荐模型。 涉及研究方向：推荐算法、推荐大模型。