• Home
  • For Entrepreneurs…
  • For Startups…
  • For Corporations…
  • Ventures
  • Listen to Podcast
  • Search Startups
  • Search Mentors
  • Browse Events

THE CENTER OF GRAVITY FOR ENTREPRENEURS IN TEXAS

Austin Deep Learning

October 4, 2022 @ 7:00 pm - 8:30 pm
RSVP FOR THIS EVENT

# LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

This paper will be presented by:
Hassan Hayat

Abstract:
Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open source our software.

Paper:
https://arxiv.org/pdf/2208.07339.pdf

Spots are limited to keep the discussions organized.

+ Google Calendar+ iCal Export

Details

Date:
October 4, 2022
Time:
7:00 pm - 8:30 pm
Website:
https://www.meetup.com/austin-deep-learning/events/288803229/

Organizer

Capital Factory
Website:
http://capitalfactory.com/

Other

RSVP Link
https://www.meetup.com/austin-deep-learning/events/288803229/

Venue

Capital Factory
701 Brazos
Austin, TX 78701 United States
+ Google Map
Website:
https://www.capitalfactory.com/
  • Mission
  • Our Team
  • Private Tours
  • Parking
  • Ask Us Anything
  • Code of Conduct
  • For Entrepreneurs…
  • For Startups…
  • For Corporations…
  • Ventures
  • We Are Hiring!
  • Media and Press

Newsletter Signup

701 Brazos St, Austin, TX 78701
(512) LIV-WORK
Crafted by HMG Creative

By Appointment Only

Our doors are open! Reach out to Members@CapitalFactory.com to book your private, in-person membership tour.

Want To Take Private Tour?

  Take a 360° Video Tour →