LLaMA: Open and Efficient Foundation Language Models (Paper Explained)

expand_more

92K views3 years ago

thumb_upLike

thumb_downDislike

library_addSave

flagReport

Yannic Kilcher check_circle

0 subscribers

#ai #meta #languagemodel LLaMA is a series of large language models from 7B to 65B parameters, trained by Meta AI. They train for longer on more data and show that something like gpt-3 can be outperformed by significantly smaller models when trained like this. Meta also releases the trained models to the research community. OUTLINE: 0:00 - Introduction & Paper Overview 4:30 - Rant on Open-Sourcing 8:05 - Training Data 12:40 - Training Hyperparameters 14:50 - Architecture Modifications 17:10 - Optimizer 19:40 - Efficient Implementation 26:15 - Main Results 38:00 - Some more completions 40:00 - Conclusion Paper: https://arxiv.org/abs/2302.13971 Website: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ Abstract: We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample Links: Homepage: https://ykilcher.com Merch: https://ykilcher.com/merch YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://ykilcher.com/discord LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon: https://www.patreon.com/yannickilcher Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

deep learning,machine learning,arxiv,explained,neural networks,ai,artificial intelligence,paper,what is deep learning,deep learning tutorial,introduction to deep learning,meta ai,meta llama,llama llm,gpt-3,large language models,transformers,chatgpt,instruction tuning,llama-i,llama paper,E5OnoYF2oAk,, Knowledge, channel_UCZHmQk67mSJgfCCTn7xBfew, video_E5OnoYF2oAk

vid:256948

Comments0

unfold_more

00:25:42

The Man behind Stable Diffusion

more_vert

Yannic Kilcher check_circle

1 views 4 years ago

00:33:56

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)

more_vert

Yannic Kilcher check_circle

1 views 5 years ago

,#nerf #neuralrendering #deeplearning View Synthesis is a tricky problem, especially when only given a sparse set of images as an input. NeRF embeds an entire scene into the weights of a feedforward neural network, trained by backpropagation through a differential volume rendering procedure, and achieves state-of-the-art view synthesis. It includes directional dependence and is able to capture fine structural details, as well as reflection effects and transparency. OUTLINE: 0:00 - Intro & Overview 4:50 - View Synthesis Task Description 5:50 - The fundamental difference to classic Deep Learning 7:00 - NeRF Core Concept 15:30 - Training the NeRF from sparse views 20:50 - Radiance Field Volume Rendering 23:20 - Resulting View Dependence 24:00 - Positional Encoding 28:00 - Hierarchical Volume Sampling 30:15 - Experimental Results 33:30 - Comments & Conclusion Paper: https://arxiv.org/abs/2003.08934 Website & Code: https://www.matthewtancik.com/nerf My Video on SIREN: https://youtu.be/Q5g3p9Zwjrk Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x,y,z) and viewing direction (θ,ϕ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons. Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yannic-kilcher Minds: https://www.minds.com/ykilcher Parler: https://parler.com/profile/YannicKilcher LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/ BiliBili: https://space.bilibili.com/1824646584 If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon: https://www.patreon.com/yannickilcher Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

00:27:33

GPT-2: Language Models are Unsupervised Multitask Learners

more_vert

Yannic Kilcher check_circle

1 views 7 years ago

keyboard_arrow_left

Tim Pool

Alex Jones Show

Bannons War Room

Trump

navigate_next

CDC, Vaccines from 6 months

more_vert

Dr. John Campbell check_circle

1 views 4 years ago

,C.D.C. Recommends 2 Covid Vaccines for under 5s ONS, https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/articles/coronaviruscovid19latestinsights/antibodies 95.9% of children aged 8 to 11 years had antibodies against SARS-CoV-2 Coronavirus (COVID-19) Update: FDA Authorizes Moderna and Pfizer-BioNTech COVID-19 Vaccines for Children Down to 6 Months of Age https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-authorizes-moderna-and-pfizer-biontech-covid-19-vaccines-children Risks of Myocarditis and Pericarditis The FDA and CDC safety surveillance systems have previously identified increased risks of myocarditis and pericarditis following vaccination with the Moderna COVID-19 Vaccine and the Pfizer-BioNTech COVID-19 Vaccine, particularly following the second dose. The observed risk is highest in, males 18 through 24 years of age for the Moderna males 12 through 17 years of age for the Pfizer-BioNTech COVID-19 Vaccine. most cases of myocarditis associated with the Moderna and Pfizer-BioNTech COVID-19 vaccines are characterized by rapid resolution of symptoms with no impact on quality of life reported by most patients who were contacted for follow-up at 90 days or more after reporting myocarditis. https://www.nytimes.com/live/2022/06/18/world/covid-19-mandates-vaccine-cases 19 million Moderna for ages 6 months through 5 years Pfizer-BioNTech ages 6 months through 4 years Dr. Rochelle P. Walensky, the C.D.C.’s director All children 6 months and older, including those who have already been infected with the coronavirus, should get a Covid vaccine Together, with science leading the charge, we have taken another important step forward in our nation’s fight against Covid-19 We know millions of parents and caregivers are eager to get their young children vaccinated, and with today’s decision, they can Albert Bourla, Chairman and Chief Executive Officer, Pfizer We know many parents in the U.S. have been eagerly awaiting an authorized vaccine for their children under 5 and we are proud to now offer them a vaccine option with a favorable safety profile Pfizer-BioNTech COVID-19 Vaccine Demonstrates Strong Immune Response, High Efficacy and Favorable Safety in Children 6 Months to Under 5 Years of Age Following Third Dose https://www.pfizer.com/news/press-release/press-release-detail/pfizer-biontech-covid-19-vaccine-demonstrates-strong-immune Vaccine efficacy was 80.3% in children 6 months to under 5 years of age. N = 1,678 had 3 doses, (omicron times) This descriptive analysis was based on 10 symptomatic COVID-19 cases identified from seven days after the third dose and accrued as of April 29, 2022. (3 in the vaccine group, 7 in the placebo group) The trial protocol specifies a formal analysis will be performed when at least 21 cases have accrued from seven days after the third dose. Final vaccine efficacy data will be shared once available. Pfizer-BioNTech COVID-19 Vaccine Receives FDA Emergency Use Authorization for Children 6 Months through 4 Years of Age https://www.pfizer.com/news/press-release/press-release-detail/pfizer-biontech-covid-19-vaccine-receives-fda-emergency-use MODERNA ANNOUNCES ITS COVID-19 VACCINE PHASE 2/3 STUDY IN CHILDREN 6 MONTHS TO UNDER 6 YEARS HAS SUCCESSFULLY MET ITS PRIMARY ENDPOINT https://investors.modernatx.com/news/news-details/2022/Moderna-Announces-its-COVID-19-Vaccine-Phase-23-Study-in-Children-6-Months-to-Under-6-Years-Has-Successfully-Met-Its-Primary-Endpoint/default.aspx 6 months to 2 years, n = 2,500 enrolled 2 to under 6, n = 4,200 enrolled Rates of fever post vaccine, greater than 38 6 months to 2 years, 17% 2 years to under 6, 14.6% 6 to 12 years, 23.9% Vaccine efficacy Similar antibody response to in 18- to 25-year-olds 6 months to 2 years, 43.7% 2 to under 6 years, 37.5% Kaiser Family Foundation survey https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-april-2022/ Plan to get under 5s vaccinated, 18% Definitely not, 27% Tom Inglesby, Johns Hopkins Center for Health Security, Bloomberg School of Public Health There’s a lot of information and trust building that needs to happen Hopefully, with time, people will have increasing confidence that it’s both effective and safe Neither vaccine was tested against BA.4 and BA.5 (exceptional immune escape) Peter Hotez, molecular virologist, National School of Tropical Medicine at Baylor College of Medicine, Houston The pediatric vaccines, won’t hold up that well in protecting against infection by the new subvariants But they will still be very effective in preventing children from going to the hospital or the intensive care unit ACIP Committee Members https://www.cdc.gov/vaccines/acip/members/index.html

00:02:26

Lerp infestation raises concerns about fire hazards in Scripps Ranch

more_vert

CBS 8 San Diego check_circle

1K views 3 years ago

00:41:07

LLaMA: Open and Efficient Foundation Language Models (Paper Explained)

more_vert

Yannic Kilcher check_circle

92K views 3 years ago

,#ai #meta #languagemodel LLaMA is a series of large language models from 7B to 65B parameters, trained by Meta AI. They train for longer on more data and show that something like gpt-3 can be outperformed by significantly smaller models when trained like this. Meta also releases the trained models to the research community. OUTLINE: 0:00 - Introduction & Paper Overview 4:30 - Rant on Open-Sourcing 8:05 - Training Data 12:40 - Training Hyperparameters 14:50 - Architecture Modifications 17:10 - Optimizer 19:40 - Efficient Implementation 26:15 - Main Results 38:00 - Some more completions 40:00 - Conclusion Paper: https://arxiv.org/abs/2302.13971 Website: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ Abstract: We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample Links: Homepage: https://ykilcher.com Merch: https://ykilcher.com/merch YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://ykilcher.com/discord LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon: https://www.patreon.com/yannickilcher Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

00:06:59

Bert Kreicher and Tom Segura Get Classy | The Roast of Tom Brady | Netflix Is A Joke

more_vert

Netflix Is A Joke check_circle

420K views 2 years ago