Writing TB Logs to S3 in PyTorch

Looks like TensorboardX supports writing to S3 and TB itself natively supports writing to S3 (anyone tried GCS?).

I am not sure whether the PyTorch writer supports S3, but judging by these threads (1 and 2) it should work if you handle your ENV variables properly:

The torch.utils.tensorboard implementation uses a writer in core TensorBoard that supports GCS and S3 if TF is installed and S3 if not installed. 

I did not know this, this is very cool!

#deep_learning

Interesting Loss Weighting Idea - Gradient Adaptive Factor

When you have 2+ losses in your NN, sometimes loss weighting is not really straightforward. Usually total loss is:

loss = loss_0 + lambda_1 * loss_1 + ...

Of course you can tune these "lambdas" manually or using some naïve NAS (or some ad hoc heuristic, i.e. this loss more important), but all these approaches have 2 drawbacks:

- Slow / compute intensive / ad hoc;
- There is no guarantee that these values are always optimal;

Usually when something is not stable (and multiple losses often explode on init) some sort of adaptive clipping is employed. I just stumbled upon a technique called Gradient Adaptive Factor, see an example here.

The idea is simple - balance your losses so that their gradient sizes are roughly similar.

#deep_learning

Last Week in AI

Most "AI" newsletters have fizzled down, became mostly unreadable noise and / or ads.

The remaining ones are usually mostly populist. Here are a couple with SNR > 0:

- https://lastweekin.ai/p/103 (by The Gradient, lol I did not know this existed)
- https://newsletter.airstreet.com/issues/your-guide-to-ai-january-2021-308710

Enjoy.

#deep_learning

PyTorch 1.8 Released

- https://pytorch.org/blog/pytorch-1.8-released/
- https://github.com/pytorch/pytorch/releases

Apart from mostly fixes, and some nice quantization (still no transformer!) and ONNX improvements, I really like this additions:

(0)

PyTorch Lite Interpreter is a streamlined version of the PyTorch runtime that can execute PyTorch programs in resource constrained devices, with reduced binary size footprint. This prototype feature reduces binary sizes by up to 70% compared to the current on-device runtime in the current release.

Link

(1)

Starting in PyTorch 1.8, we have added support for ROCm wheels providing an easy onboarding to using AMD GPUs.

Link

(2)
New beta benchmark utils
Link

(3)
New PyTorch Mobile demos

(4)
New quantization API
link

(5)
New related libraries release (i.e. torchaudio, torchvision), looks like they are tied to PyTorch releases now

#deep_learning

Torch FX

- https://pytorch.org/docs/master/fx.html

X is a toolkit for developers to use to transform nn.Module instances. FX consists of three main components: a symbolic tracer, an intermediate representation, and Python code generation.

I understand that people building PyTorch usually favour flexible toolkits (and they expose a lot to an end user) and most likely they just realized that static quantization was too complex for an average user to handle and they wrote this as an engine for automated quantization transformations, which is cool. Designing a proper API is always a balancing act.

Over the years, I became quite good in monkey patching PyTorch code just using python's and pytorch tools (e.g. module.named_modules()). So I wonder what the killer use case of this feature would be?

One thing comes to mind immediately - when you have the same models with static control flows and you need to create a quantized / torch script version of it. Now it is a pain in the ass - because it requires manually switching them back and forth (switch on, create a quantized TorchScript version one, switch back, create another one, etc).

Will I use it? I guess I need to sleep on it. We ended up not using static quantization very much. Looks very cool and flexible, serves a real purpose, but usually stupid one line hacks can do the same without learning a new tool.

So idk, what do you think? Do you like any of the examples? I like the invert one.

#deep_learning

New Benchmarking Tool in PyTorch

https://pytorch.org/tutorials/recipes/recipes/benchmark.html#pytorch-benchmark

Looks a bit over-complicated at the first glance (why provide classes for random tensor generation, I have no idea), but it has a few very nice features:

- Automated num_threads handling
- Automated CUDA synchronization
- Report generation, storing the results, comparing the results

But I suppose there is nothing wrong just using %%timeit manually setting num_threads.

#deep_learning

Building Your Own Supercomputer Cheap (RU)


My guest post on ODS @ habr:

- https://habr.com/ru/company/ods/blog/546808/

EDIT - some awesome comments!

#deep_learning

📎

BIFURCATED RISER X16 TO 2X8 (SET)

Remember that there is a very limited number of motherboards with 5+ PCIE slots?

Now there are risers like this - https://riser.maxcloudon.com/ru/bifurcated-risers/25-bifurcated-riser-x16-to-2x8-set.html

Has anyone tried something similar for DL?

#deep_learning

📎

While ThreadRipper Pro MBs are impossible to buy, this MB may be the base for our next huge server build:

- https://market.yandex.ru/product--materinskaia-plata-asrock-rack-romed8-2t/705623617

Looks a bit expensive (and it uses ECC RAM + EPYC processors), but with 7 x PCIE 4.0 16x and 2x10Gbit/s Ethernet possibilities are limitless.

📎

And another hack - buying used 100 GBit/s infiniband cards from ebay, they are cheap now in the US

#deep_learning

PyTorch + AMD Inference

We were benchmarking our networks on Intel vs AMD processors using out-of-the-box official build.

And Intel mostly is better (with the same number of threads and roughly the same core speed and lack of overclocking). I was wondering why this is, and then I found this thread.

To be honest I have little motivation to invest time in redoing our environment builds from scratch with OpenBLAS + CUDA (and most likely it is not worth the time since in production most likely there will be Intel CPUs).

But I wonder, does anyone in the community have dockerized dev environment builds based around CUDA + OpenBLAS? Because looks like out of the box PyTorch ships with MKL by Intel.

#deep_learning

Factorized Networks

I really like the idea from this article - https://www.microsoft.com/en-us/research/blog/factorized-layers-revisited-compressing-deep-networks-without-playing-the-lottery/

Basically you do not prune networks (which does not readily transfer into inference) or distill your teacher network into a student, but train a low-rank factorized version of the network with some optimizations from scratch.

This article even has code, but basically ... this is an older fork of fairseq imported via 1 commit. So good luck doing what authors did not bother to do (providing a stand-alone implementation).

So the question is - has anyone seen a minimalist stand-alone implementation for something similar?

#deep_learning

Finally Proper GPU Support in Compose!

It happened finally (some time ago, I just checked now)!

Now this obsolete runtime: nvidia syntax can be replaced with this more versatile syntax:

    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]

This together with CUDA_VISIBLE_DEVICES gives you full control of your GPU environment within compose.

https://docs.docker.com/compose/gpu-support/

#deep_learning

MLPerf Inference v1.0

- Inference Edge v1.0 https://mlcommons.org/en/inference-edge-10/
- Inference Datacenter v1.0 https://mlcommons.org/en/inference-datacenter-10/

The immediate conclusion (as expected) - enterprise kinky party. The second conclusion - they mostly compare vastly different systems (mostly HPC), which is good.

Honestly I do not really care for A100 vs. A?0 vs. Quadro vs. T4, but edge benchmarks are always rare and nice.

The most interesting spreadsheet IMO is this one.

And here I see some quite interesting stuff:

- Firefly-RK3399 looks similar to RPI 4 in performance (has anyone used it?)
- NVIDIA Jetson AGX Xavier looks ~2x faster than both of them (and probably is much more expensive and unobtainable)
- TFLite / ArmNN - but no ONNX or PyTorch on ARM, I wonder why
- int8 very much a must-have on these devices, I see performance boosts up to 2x

PS

Firefly-RK3399 has a PCIE M2 slot, so theoretically you can plug in PCIE accelerator sticks there? =)
It also runs on Ubuntu?

#hardware
#deep_learning

Nice Transformer-based S2S Example

Minimalistic clean code is hard to come by nowadays:

- https://twitter.com/full_stack_dl/status/1349156930518859780

Transformer S2S decoding is not often properly explained.

Tldr - this enables fast masked training, but inference is step-by-step, no miracle here.

#deep_learning

Einops and Einsum in PyTorch

Previously there was an attempt of making DL code more readable via named tensors (still a prototype, they "imported" a third party library). A cool idea, but I have never really seen anyone using it (me too).

Now a similar thing with (not) new Einstein notation for Deep learning:

- https://pytorch.org/docs/stable/generated/torch.einsum.html
- https://stackoverflow.com/questions/55894693/understanding-pytorch-einsum
- https://github.com/arogozhnikov/einops

Will it stick? No idea. Einsum may be a blessing for some complex code. It is not necessarily more readable generally if you got used to basic APIs (like bmm for example).

Also I believe it may be adapted into PyTorch as syntactic sugar.

#deep_learning

Some Mobile DNN Inference Benches

https://ai-benchmark.com/ranking.html
https://ai-benchmark.com/ranking_detailed.html
https://mlcommons.org/en/inference-mobile-10/

TLDR

- Using GPUs is 3-10x faster
- All of them test with TF / native APIs
- Expensive phones are faster (orly)
- No PyTorch (lite) accelerated results yet

#deep_learning

Huge Image + Text Contrastive Representation Learning

Typically, huge data + compute does not equal research news. People also are always over-excited about GANs and trillion parameter sized networks for some reason that eludes me. Yet the most useful "huge" compute networks are not shared (for obvious reasons - because they are useful in real life).

This is a bit different - looks like Google scraped 1.8B images with their text annotations and applied contrastive dual-encoder learning. The networks are large, but not needlessly so (they are small enough to fit on one conventional GPU?).

If you interpret posts by MS / FAIR / Google as "they throw all the shit and ton of compute at the wall and see what sticks", probably this is the minimum scale on which this task kind of works (there were some similar attempts by FAIR).

What impresses me though is real zero-shot performance on Imagenet, i.e. not fake academic "zero-shot" performance when networks are pre-trained on 10-100x sized datasets and then tuned on 5-10% of Imagenet, but true "zero-shot" performance.

This kind of points at the compute required (typically you can divide compute requirements by Google by 10, but still) :

The learning rate is warmed up linearly to 1e-3 from zero in 10k steps, and then linearly decay to zero in 1.2M steps (∼12 epochs). We train the model on 1024 Cloud TPUv3 cores with 16 positive pairs on each core. Therefore the total effective batch size is 16384.
5.1.

Also I could not find any mention of public weights. Google usually does not publish huge compute networks ... that are actually useful in production.

https://ai.googleblog.com/2021/05/align-scaling-up-visual-and-vision.html

I also remember the bs hype wave about OpenAI large transformer ... but Google published a similar network - and no one cared. Looks this kind of answers if those networks were useful.

#deep_learning

PyTorch 1.9 Released

https://pytorch.org/blog/pytorch-1.9-released/
https://github.com/pytorch/audio/releases/tag/v0.9.0

Major improvements to support scientific computing, including torch.linalg, torch.special, and Complex Autograd

Major improvements in on-device binary size with Mobile Interpreter (!)

Native support for elastic-fault tolerance training through the upstreaming of TorchElastic into PyTorch Core

Major updates to the PyTorch RPC framework to support large scale distributed training with GPU support

New APIs to optimize performance and packaging for model inference deployment (!)

Support for Distributed training, GPU utilization and SM efficiency in the PyTorch Profiler

#deep_learning

Transformer Module Optimization

Article on how to apply different methods to make your transformer network up to 10x smaller and faster:

- Plain model optimization and PyTorch tricks;
- How and why to use FFT instead of self-attention;
- Model Factorization and quantization;

https://habr.com/ru/post/563778/

#deep_learning

Embedding Quantization in PyTorch

Currently the following modules are supported by the following quantization modes:

- CNN (i.e. nn.Conv) => static quantization (you need to store statistics);
- RNN or Transformer (i.e. nn.Linear) => dynamic (you just convert weights);

Looks like less love was given to the Embedding layers, because they require static quantization: https://github.com/pytorch/pytorch/issues/65185

In any case, someone tested it and provided a simple recipe, which is absent from docs.

#deep_learning