Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

🖥 Github: https://github.com/kaistmm/SSLalignment

📕 Paper: https://arxiv.org/abs/2407.13676v1

🚀 Dataset: https://paperswithcode.com/dataset/is3-interactive-synthetic-sound-source

https://t.me/DataScienceT ⭐

🎭 TRG: new SOTA in 6DoF Head 🎭

👉 ECE (Korea) unveils TRG, a novel landmark-based method for estimating a 6DoF head pose which stands out for its explicit bidirectional interaction structure. Experiments on ARKitFace & BIWI confirm it's the new SOTA. Source Code & Models to be released 💙

🤣 Review: https://t.ly/lOIRA

🤣 Paper: https://lnkd.in/dCWEwNyF

😏 Code: https://lnkd.in/dzRrwKBD

https://t.me/DataScienceT ⭐

Neural General Circulation Models for Weather and Climate

General circulation models (GCMs) are the foundation of weather and climate prediction. GCMs are physics-based simulators which combine a numerical solver for large-scale dynamics with tuned representations for small-scale processes such as cloud formation. Recently, machine learning (ML) models trained on reanalysis data achieved comparable or better skill than GCMs for deterministic weather forecasting. However, these models have not demonstrated improved ensemble forecasts, or shown sufficient stability for long-term weather and climate simulations. Here we present the first GCM that combines a differentiable solver for atmospheric dynamics with ML components, and show that it can generate forecasts of deterministic weather, ensemble weather and climate on par with the best ML and physics-based methods. NeuralGCM is competitive with ML models for 1-10 day forecasts, and with the European Centre for Medium-Range Weather Forecasts ensemble prediction for 1-15 day forecasts. With prescribed sea surface temperature, NeuralGCM can accurately track climate metrics such as global mean temperature for multiple decades, and climate forecasts with 140 km resolution exhibit emergent phenomena such as realistic frequency and trajectories of tropical cyclones. For both weather and climate, our approach offers orders of magnitude computational savings over conventional GCMs. Our results show that end-to-end deep learning is compatible with tasks performed by conventional GCMs, and can enhance the large-scale physical simulations that are essential for understanding and predicting the Earth system.

Paper: https://arxiv.org/pdf/2311.07222v3.pdf

Code: https://github.com/google-research/neuralgcm

Code: https://github.com/google-research/dinosaur

https://t.me/DataScienceT ⭐

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Portrait Animation aims to synthesize a lifelike video from a single source image, using it as an appearance reference, with motion (i.e., facial expressions and head pose) derived from a driving video, audio, text, or generation. Instead of following mainstream diffusion-based methods, we explore and extend the potential of the implicit-keypoint-based framework, which effectively balances computational efficiency and controllability. Building upon this, we develop a video-driven portrait animation framework named LivePortrait with a focus on better generalization, controllability, and efficiency for practical usage. To enhance the generation quality and generalization ability, we scale up the training data to about 69 million high-quality frames, adopt a mixed image-video training strategy, upgrade the network architecture, and design better motion transformation and optimization objectives. Additionally, we discover that compact implicit keypoints can effectively represent a kind of blendshapes and meticulously propose a stitching and two retargeting modules, which utilize a small MLP with negligible computational overhead, to enhance the controllability.

page: https://liveportrait.github.io/

paper: https://arxiv.org/abs/2407.03168

code: https://github.com/KwaiVGI/LivePortrait

jupyter: https://github.com/camenduru/LivePortrait-jupyter

https://t.me/DataScienceT ⭐️

Guess what? There's a chance to snag 10 million $Whale tokens! 🐳

And the best part? You can get them for FREE right now! 🎁

Here’s how:

1️⃣ Share your referral link from the "Earn" section at @Whale.
2️⃣ Every USD you earn equals a $Whale token.
3️⃣ Earn cash bonuses for every friend you invite!

Get your tokens before they’re all gone! 🥰

⚡️⚡️ SAM v2 is out! ❤️‍🔥❤️‍🔥

✅ #Meta announced SAM 2, the novel unified model for real-time promptable segmentation in images and videos. 6x faster, it's the new SOTA by a large margin. Source Code, Dataset, Models & Demo released under permissive licenses 💙

🔗 Review: https://t.ly/oovJZ

✅ Paper: https://t.ly/sCxMY

✅ Demo: https://sam2.metademolab.com

🔥 Project: ai.meta.com/blog/segment-anything-2/

⭐️ Models: github.com/facebookresearch/segment-anything-2

https://t.me/DataScienceT 🗣

🖥 StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset.

🖥 Github: https://github.com/huochf/StackFLOW

📕 Paper: https://arxiv.org/abs/2407.20545v1

🚀 Dataset: https://paperswithcode.com/dataset/behave

https://t.me/DataScienceT ⭐️