The Open Data Science Conference West 2023 took place in San Francisco between October 30th and November 2nd. The event hosted over 250 speakers who focused on AI, data science, and the future of big data.
CodiLime was represented during this event by our Director of Engineering, Maciej Manturewicz, Head of Data Engineering, Tomasz Jędrośka, and Business Development Manager, Katarzyna Mirońska-Veigas.
Below, you can find Tomasz’s summary and impressions of the OSDC West 2023. Dive in to discover insights from the world’s leading data science conference.
Intro and session evaluation methodology
Last week, together with my team I had the chance to attend ODSC West 2023 - one of the most well-known data science conferences in the US with speakers from pretty much all major AI-focused companies as well as university lecturers and scientists. My goal was to access new expertise from industry leaders to benefit the projects CodiLime is engaged in, and find out whether our engineering teams would be able to offer meaningful synergies in their engagement. As it turned out, the ODSC venue made both those goals possible to achieve.
First, let me briefly describe the conference and my methodology for defining and scoring the sessions I attended.
During three days (31.10 - 2.11) more than 100 in-person sessions were held; I took part in nine (+ one panel talk). Sessions were divided into categories - talks, tutorials, and workshops, however from my observations each lecturer approached their session individually, and many talks had a demo code to work with included in the material. Having said that, I’m describing and summarizing them all using the same methodology.
Before we jump to the juicy part, you need to be aware of the hyperparameters I’m tuned with - I favor people who teach, not lecture. I enjoy sessions when the leader interacts with the audience, makes sure the pace of the session is in line with the attendee's perception and lastly, a good meme or joke from time to time is very appreciated so that a spark of joy lightens the dark corridors of mathematical and statistical knowledge. Apart from the above, I also pay attention to whether the material is well prepared (including the presented slide deck), the speaker talks fluently, and whether they can quickly adapt when the audience asks to focus on a particular topic.
I have summarized each session with a score from 1 to 10 indicating how much I liked it and how useful it was in my opinion as well as a Clap MeterTM indicator describing how the audience appreciated the session (no I didn’t have an audio recording device and an evaluation model calculating the score; I used a good old analog method and just listened to the volume and length of the applause - so nineties, I know…)
Alright, after this probably-much-too-long disclaimer, without further ado, let's jump to the interesting part.
ODSC ‘23 Sessions in chronological order
Learn how to Efficiently Build and Operationalize Time Series Models in 2023 by Jelle Vlassenbroeck (Tangent Works), and Simon Mestdagh (Tangent Works)
The lecture addressed the challenges of operationalizing time series (TS) analytics due to the large volume of data involved. The presenters introduced a new approach to TS analytics that can handle large datasets and perform both forecasting and anomaly detection. They emphasized the importance of data quality and the need for models to adapt to real-world conditions and behaviors.
The presenters also showcased their tool, TIM, which utilizes a novel engine to discover key features, reduce dimensionality, and create relevant prediction models. A live demo demonstrated the tool's capabilities in identifying important features, adapting to changing data patterns, and building models rapidly. It was emphasized that the tool automatically reduces the amount of features paying attention to orthogonalities and detecting which properties are the most significant to the final output. Although there was a slight misalignment in the presentation deck, the session demonstrated TIM's capabilities effectively. The tool offered crucial features, showcased how these features might change over time, and automatically adjusted historical data relevance, even considering events like the onset of COVID-19.
Overall, the session provided valuable insights into the challenges and solutions for operationalizing TS analytics. In short, it was a rather product-oriented lecture delivered straightforwardly without major fireworks.
Session score: 6/10
Clap Meter: 60%
Building LLM-powered Knowledge Workers over Your Data with LlamaIndex by Jerry Liu (LlamaIndex)
During this session, Jerry delved into the practical aspects of implementing RAG (retrieval-augmented generation) in production. The presentation focused on LlamaIndex, a data framework for large language model (LLM) applications, covering topics like ingestion, indexing, querying, and orchestration of LLMs with vector and graph databases. The flow was explained from a foundation model plus extending document input to response output, emphasizing the importance of chunking and generating embeddings for effective querying. A nice cherry on top for me was mentioning the Ray framework (an open-source library that provides the compute layer for parallel processing) and presenting relevant examples of how it speeds up the whole process.
The presenter, well-prepared with clear slides and demo materials, navigated complex concepts with precision and consistency. Audience interaction was encouraged, with the speaker addressing questions and explaining intricacies during breaks. However, some attendees seemed puzzled, leading to a few departures. The session delved into evaluating LLMs and RAG comprehensively, discussing metrics and hit rates for model assessment. The speaker emphasized the significance of evaluating responses and demonstrated the use of LLamaIndex CorrectnessEvaluator for quality assessment.
Although the content was substantial, the presenter's rapid speech occasionally made comprehension challenging. Despite this, the session provided valuable insights into building effective RAG pipelines and evaluating their performance, engaging the audience in discussions and practical demonstrations.
Session score: 7/10
Clap Meter: 60%
Identifying the Next Generation of AI Startups. A panel discussion by Vasundhara Chetluru (Tiffin), Anne Dwane (Village Global), Igor Tabber (Corical Ventures), and Akshay Bhushan (Tola Capital)
The panel session on "Identifying the Next Generation of AI Startups" at the ODSC West conference explored the evolving landscape of AI trends and their impact on the market. The participants highlighted the current AI hype cycle, emphasizing the importance of focusing on solving real problems rather than just chasing trends. The discussion focused on innovative AI applications, including startups like Auba (improving supply chains with AI), and Arcus (focusing on demand forecasting using a broader context with third-party data).
The speakers also addressed questions about challenges related to AI regulation saying model certification might be a way forward. When asked about innovation they emphasized that the founders need to focus mostly on customers’ needs.
The participants stressed the importance of engaged founders in AI startups, highlighting qualities such as confidence, resilience, and having a good imagination regarding the future product’s market fit. The session also touched on disruptive innovations, business models for AI, and the hope for more meaningful AI-generated (or AI-curated) content, reversing the current trend of the Internet ‘being increasingly more stupid’.
In terms of the overall impressions of the session, Igor provided intelligent and humorous answers, engaging the audience effectively. Anne summarized the discussions conclusively, while Akshay maintained focus and precision. Even though I’m not a huge fan of panel talks, this one was well-prepared, and aimed to educate attendees about the practical aspects of AI innovation, encouraging us as participants to focus on being better, faster, and cheaper in our AI initiatives.
Session score: - (panel discussion)
Clap Meter: 70%
Troubleshooting Large Language Models in Production with Embeddings and Evals by Amber Roberts (Arize AI)
Amber delivered a compelling session titled "Troubleshooting LLMs in Production with Embeddings and Evals." She vigorously initiated the talk, establishing a strong connection with the audience through engaging and witty questions. She introduced tools that address effective switching of the prompt template to improve the RAG (retrieval-augmented generation) outcomes (Phoenix) as well as implement evaluation and observability for each RAG section (Arize). Amber addressed significant challenges, particularly LLM hallucinations, and demonstrated how Phonix and Arize offer solutions throughout the RAG process.
Amber also delved into common LLM improvement workflows, citing the e-commerce and travel sectors as examples. She explored various evaluation metrics, including LLM-assisted evaluation, user feedback, and task-based metrics, while emphasizing integration challenges, confusion, and lack of rigor that might affect the interpreted value of those metrics. The presenter skillfully compared traditional methods with innovative approaches, illustrating for example, how modifying prompt templates can enhance experiment performance using Phoenix.
Throughout the session, Amber actively engaged with the audience, addressing their use cases and maintaining a fluent, engaging delivery. She presented Arize & Phoenix, offering insights into the RAG callback system, testing, and metrics, highlighting metrics beyond traditional measurements such as latency, token usage, and prompt templates. The session flowed smoothly, with an optimistic and witty style keeping attendees engaged. She concluded by outlining actionable next steps for implementing the discussed tools, emphasizing the importance of monitoring and alerting based on the gathered metrics. The presentation was well-prepared, and the slides were readable and accurate, all in all, one of the best sessions of this year’s ODSC.
Session score: 8/10
Clap Meter: 85%
How to Deliver Contextually Accurate LLMs by Jake Bengtson (Cloudera)
Jake delivered a presentation on various applications of generative AI, such as text-to-text and text-to-speech, emphasizing their business applications like developer productivity, call center automation, and document services. He highlighted the evolution in this field over the past year since OpenAI's inception and discussed Cloudera's use cases, including local fine-tuned GPT models and enterprise-specific solutions. Jake explored the challenges faced, such as privacy, accuracy, and the black-box nature of AI, proposing solutions like hosted open-source language models connected to enterprise databases.
Jake discussed different approaches to large language model (LLM) implementation, including prompts, RAG, parameter-efficient fine-tuning and full fine-tuning. He presented Applied ML Prototypes (AMP) as best practices for quick deployment and explained an open-source chatbot augmented with corporate data, showcasing a live demo with contextually augmented flow. He covered fine-tuning foundation models, comparing full parameter fine-tuning with adapters tailored for specific use cases, highlighting the flexibility and reduced computational requirements of adapters.
A customer use case from OCBC Bank illustrated the rapid application deployment, taking only three days from idea to implementation, utilizing Ray Serve 2.4. Jake demonstrated Cloudera's complete AI stack, using the bloom1B1 model. Although he skipped some questions at the end, his presentation was generally well-received. He navigated through the material fluently, maintaining a good pace and a straightforward flow. The presentation was engaging, demonstrating live examples despite a few non-functional links, which Jake resolved by browsing the source. Overall, the session was informative, well-prepared, and focused on teaching attendees about the material rather than merely reading from the slides.
Session score: 7/10
Clap Meter: 75%
Anomaly Detection for CRM Production Data by Tuli Nivas, Geeta Shankar (Salesforce)
Tuli Nivas and Geeta Shankar hosted a session on Salesforce's performance analysis, focusing on actionable insights derived from log data, trace data, and APM analysis. Tuli initiated the talk, providing an overview of Salesforce's processes and challenges in customer experience. While she presented actual data highlighting the approaches toward preventing poor server performance, her introduction felt somewhat rehearsed as if she was reading from a script.
Geeta Shankar took over, demonstrating a bit more fluency and engaging the audience with a demo. She shared a GitHub repository for reference and delved into performance engineering concepts, detailing Salesforce’s stack components - pods and caching strategy. Despite minor issues with slide readability, Geeta swiftly addressed them, showcasing code examples and explaining the process step by step. Geeta explored anomaly detection using linear regression, interacting with the audience and addressing questions smoothly. She employed the Prophet framework for time series analysis, explaining configurations and metrics, although some explanations lacked depth, leaving out the 'why' and actionable results (that were actually emphasized as crucial by both lecturers).
The session covered topics like customer growth analysis and Apex code monitoring, although a bit too simplistically for my taste. Tuli returned to discuss future plans, including integrating outcomes with Tableau and even ‘more real-time’ anomaly detection. While Geeta's segment was more fluid, the cases presented were simplistic, earning the session an overall average score. The speakers managed to engage the audience effectively, although some parts felt rehearsed or lacked in-depth explanations, impacting the session's overall effectiveness.
Session score: 5/10
Clap Meter: 55%
Scope of LLMs and GPT Models in the Security Domain by Nirmal Budhathoki (Microsoft)
Nirmal jumped into the session by describing the evolution of IT threats, emphasizing the importance of prioritizing security ahead of product development. He highlighted the rising trend of social engineering attacks, particularly phishing, and discussed various past vulnerabilities in popular platforms like H2O Flow and Keras, pointing out that one should be aware that not paying attention to security might lead to sensitive information disclosure and model theft. The presenter showcased the significance of data scientists' awareness as ‘data scientists are the new administrators’ due to their extensive data access.
The speaker presented insights from a survey conducted at Microsoft, revealing IT security areas where large language models (LLMs) could benefit, such as incident management and threat intelligence, while emphasizing challenges related to responsible AI and trust in LLMs. He introduced GenAI's potential, including efficiency in incident management and automation to combat zero-day attacks.
The session featured a discussion on the importance of responsible AI, illustrated with real-life examples, including an incident involving Samsung employees and ChatGPT (using the chatbot outputs with fact-checking). The presenter explained techniques to trick AI models into revealing potentially harmful information, stressing the need for regulation in the field. While the presenter's English and reasoning were clear, the session lacked audience engagement, with limited interactions and minimal humor. Despite this, the presenter effectively addressed deepening questions with real-life examples, providing valuable insights into the crucial realm of AI security and regulation.
Session score: 6.5/10
Clap Meter: 40%
Why did my AI do that? Decoding Decision-making in Machine Learning by Swagata Ashwani (Boomi)
Swagata prepared for us a talk that commenced with a playful inquiry and memes about peculiar responses from ChatGPT, followed by a brief introduction to large language models (LLMs). I enjoyed the fact that the lecturer decided to narrow down this part since it had already been covered multiple times in other sessions. The presentation smoothly transitioned to an overview of GenAI, offering concise yet insightful real-life examples.
The core focus of the talk revolved around the black box dilemma in AI, emphasizing its significance in fields like law and medicine where AI consumers need a thorough explanation of why and how answers are formulated. Ashwani explored the concept of explainability, discussing the extent to which we can comprehend AI decision-making processes. Although the explanation was solid, the accompanying slides were somewhat lacking. Afterward, the session delved into various approaches, including SHAP (based on game theory) and LIME (local interoperable model-agnostic explanations). Swagata smoothly illustrated these solutions, offering clear and meaningful examples, and also suggesting particular tools for adequate use cases (SHAP for finding global, and LIME for local explanations; use of LIME was also recommended in ad-hoc cases due to it being less compute-expensive)
The presenter's fluency and engagement with the audience were commendable, contributing to a comfortable atmosphere during the session. Even though it was a ‘talk’, Ashwani efficiently utilized a demo, simplifying the presented ideas using a classical ML model, although it was emphasized that neural networks would eventually be more suitable. Overall, Swagata’s session effectively balanced theoretical explanations with practical demonstrations, ensuring attendees grasped the material, rather than simply reading the slides.
Session score: 8/10
Clap Meter: 90% (unfortunately, I needed to leave early for a customer meeting, but I took into account other participants' comments)
Beyond the Buzz: Decoding Popularity Bias in Large-scale Recommender Systems by Amey Porobo Dharwadker (Meta)
Amey's presentation delivered an analysis focused on the complexities of recommendation systems, highlighting the challenges posed by popularity bias. He explained the phenomena of over-recommending popular content, leading to limited content discovery and user engagement. The lecture was structured, starting with an introduction to recommendation systems as multi-sided marketplaces and their impact on user interactions (the less content they engage with, the sooner they churn from the platform).
Even though the session was performed in a lecture style with limited audience interaction, Amey demonstrated a deep understanding of the topic. He discussed the negative effects of popularity bias and proposed various mitigation strategies. These included pre-processing techniques such as data sampling and filtering, in-processing modifications to recommendation models, and post-processing methods like score adjustment and rank aggregation.
The presenter used a simple deck, and although there were minor gaps in the slides, the audience found the topic intriguing. Despite the basic presentation style, attendees engaged actively, asking numerous questions about improving content ranks and system workings. Amey emphasized the need for further research, public datasets, and enhanced experimental approaches to tackle bias in recommendation systems effectively. While the session lacked extensive audience interaction, it provided valuable insights into mitigating popularity bias, making it a notable talk at the conference.
Session score: 6/10
Clap Meter: 65% (75% after the Q&A session, which was the longest and most engaging of all)
Aligning Open-source LLMs Using Reinforcement Learning from Feedback by Sinan Ozdemir (LoopGenius)
Sinan Ozdemir's session was a captivating presentation that delved into the intricate process of aligning large language models (LLMs). Sinan, with a dynamic and engaging presentation style akin to a super energetic stand-up comedian, started by explaining the concept of alignment and its significance in LLMs. He discussed real-world cases, including aligning FLAN-T5 summaries for better grammar and incorporating human feedback loops to enhance responses.
Sinan demonstrated preparedness by navigating through complex technical aspects, including reinforcement learning (RL) techniques. He introduced the audience to a simplified RL process, emphasizing the importance of understanding over mere execution. Interactive coding sessions showcased his expertise, ensuring attendees grasped the key concepts. Sinan encouraged questions and emphasized a teaching approach over lecturing.
The presenter spotlighted crucial aspects of RL, such as reward systems and efficient fine-tuning techniques. He explained the nuances of the training process, addressing challenges with picking the correct hyperparameters and the rationale behind each choice with depth. Sinan highlighted the significance of checking semantic similarity between questions and answers as well as questions with the provided content, showcasing the practical application of the approaches presented.
The slides were readable and correct, enhancing the overall learning experience. Sinan's session stood out for its audience engagement, fluency, and focus on teaching rather than just reading from slides. The session's highlight was Sinan's ability to make complex topics accessible, creating an informative and interactive atmosphere, making it in my view the best talk of the ODSC ’23 conference. Sinan was promoting his new book, Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs 
. I must say, if he writes at least half as well as he teaches you should definitely get it!
Session score: 10/10
Clap Meter: 90%
Summary
That’s it! That was my ODSC ’23 conference in a nutshell. Hopefully, the summary was fruitful for you and indicated which session you might find interesting enough to google its author and/or the presented material (or attend a similar session during the next ODSC edition.) This material has its obvious limitations - the conference had a lot more sessions happening in parallel and there were also virtual sessions which I’ll probably look into soon.
Should you have any questions regarding the material feel free to reach out via LinkedIn or email, I’m happy to address them and share more insights. Overall the event turned out to be very interesting and I already look forward to the next edition - hopefully, see you there!
