CNNs são capazes de segmentar objetos com base no reconhecimento desses mesmos objetos.

Segmentação Semântica

Tutoriais Gerais

Estrutura geral de uma rede para segmentação semântica:

1. CV-Tricks: Image Segmentation using deconvolution layer in Tensorflow
2. Medium: How to do Semantic Segmentation using Deep learning
3. Para começar! Keras semantic-segmentation-example

Segmentação Semântica com Keras e Theano

Os tutoriais desta seção usam Theano e não TensorFlow como backend… É bem mais simples de instalar e fazer rodar na sua GPU.

1. Image Segmentation Keras : Implementation of Segnet, FCN, UNet and other models in Keras
   1. Observação: O código postado neste Git é muito claro e simples e ajuda bastante a compreender como funcionam os modelos. Ele utiliza arquivos de pesos mais antigos do Keras para as redes que servem para os dois braços do “U” nas redes Unet e SegNet, então tem de ser baixado manualmente (se você usar Keras.Applications isto é feito automaticamente, mas exige um trabalho de adaptação)
   2. Instalando Theano: http://deeplearning.net/software/theano/install.html

Modelos Clássicos

Unet e TernausNet

Unet atinge resultados muito bons em várias aplicações de segmentação biomédica diferentes. Em seu artigo, Ronneberger et al. (2015) usaram bastante a prática de data augmentation, que foi a principal técnica que fez com que obtivessem bons resultados, já que seu conjunto de dados era pequeno.

A parte inicial da rede encoder, é igual a uma arquitetura típica de uma rede convolucional. Entretanto, não possui as camadas totalmente conectadas. A modificação na arquitetura feita pela Unet, consiste em ter um grande número de canais com caraterísticas extraídas também na etapa de upsampling, como pode ser visto na figura. Isso permite que a rede consiga propagar mais informações sobre contexto da imagem às camadas que estão “acima”, ou seja, de maior resolução. Outras características importantes sobre a Unet são mostradas na tabela adiante, ao final desta seção.

Seus resultados em desafios de segmentação, como por exemplo, o ISBI cell tracking challenge 2015, se mostraram superiores a outras redes em dois dos mais desafiadores datasets.

Links: Unet e TernausNet

1. Modelo original da Univ. Freiburg: U-Net: Convolutional Networks for Biomedical Image Segmentation – https://arxiv.org/abs/1505.04597
   1. https://en.wikipedia.org/wiki/U-Net
2. Modelo melhorado do MIT: TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation – https://arxiv.org/abs/1801.05746
   1. Git: https://github.com/ternaus/TernausNet
   2. V2 do modelo: https://github.com/ternaus/TernausNetV2
3. Link com vários modelos: Divam Gupta – Image Segmentation Keras : Implementation of Segnet, FCN, UNet and other models in Keras
4. TowardsDataScience::Understanding Semantic Segmentation with UNET
5. Jeremy Jordan::An overview of semantic image segmentation
6. Analytics Vidhya::Semantic Segmentation: Introduction to the Deep Learning Technique Behind Google Pixel’s Camera!
7. Nanonets::How to do Semantic Segmentation using Deep learning
8. Outras implementações de UNet e TernausNet:
   1. https://github.com/zhixuhao/unet [Keras]
   2. https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/ [Caffe + Matlab]
   3. https://github.com/jocicmarko/ultrasound-nerve-segmentation [Keras]
   4. https://github.com/EdwardTyantov/ultrasound-nerve-segmentation [Keras]
   5. https://github.com/ZFTurbo/ZF_UNET_224_Pretrained_Model [Keras]
   6. https://github.com/yihui-he/u-net [Keras]
   7. https://github.com/jakeret/tf_unet [Tensorflow]
   8. https://github.com/DLTK/DLTK/blob/master/examples/Toy_segmentation/simple_dltk_unet.ipynb [Tensorflow]
   9. https://github.com/ZijunDeng/pytorch-semantic-segmentation [PyTorch]
   10. https://github.com/akirasosa/mobile-semantic-segmentation [Keras]
   11. https://github.com/orobix/retina-unet [Keras]
   12. https://github.com/masahi/nnvm-vision-demo/blob/master/unet_segmentation.py [onnx+nnvm]
   13. https://github.com/qureai/ultrasound-nerve-segmentation-using-torchnet [Torch]
   14. https://github.com/ternaus/TernausNet [PyTorch]

SegNet

SegNet consists of layers called encoders and decoders. Each encoder applies convolution, batch normalization and a non linearity, then applies max pooling on the result, while storing the index of the value extracted from each window. Decoders are similar to the encoders, the difference is that they don’t have a non linearity, and they upsample their input, using indices stored from the encoding stage. After the final decoder, the output is fed to a softmax classifier which gives the final prediction. The prediction is going to be an n channel image so we have to write a separate function to convert it into an RGB image where we can look at the results qualitatively.

Novidades na Arquitetura de SegNet

Uses a novel technique to upsample encoder output which involves storing the max-pooling indices used in pooling layer. This gives reasonably good performance and is space efficient.

VGG16 with only forward connections and non trainable layers is used as ÷encoder. This leads to very less parameters.

Links:

1. Página do Projeto: SegNet
2. Alex Kendall, Vijay Badrinarayanan and Roberto Cipolla “Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding.” arXiv preprint arXiv:1511.02680, 2015 ( .pdf )
3. Vijay Badrinarayanan, Alex Kendall and Roberto Cipolla “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.” PAMI, 2017 ( arxiv .pdf ) ( poster )
4. Vijay Badrinarayanan, Ankur Handa and Roberto Cipolla “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling.” arXiv preprint arXiv:1505.07293, 2015 ( .pdf )
5. Link com vários modelos: Divam Gupta – Image Segmentation Keras : Implementation of Segnet, FCN, UNet and other models in Keras
6. Alex Kendall: Implementation of SegNet in Caffe: A Deep Convolutional Encoder-Decoder Architecture for Semantic Pixel-Wise Labelling
7. Alex Kendall: SegNet-Tutorial
8. Siddhant Kumar: Summary of – SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
9. Outras implementações:
   1. https://github.com/developmentseed/caffe/tree/segnet-multi-gpu [Caffe]
   2. https://github.com/preddy5/segnet [Keras]
   3. https://github.com/imlab-uiip/keras-segnet [Keras]
   4. https://github.com/andreaazzini/segnet [Tensorflow]
   5. https://github.com/fedor-chervinskii/segnet-torch [Torch]
   6. https://github.com/0bserver07/Keras-SegNet-Basic [Keras]
   7. https://github.com/tkuanlun350/Tensorflow-SegNet [Tensorflow]
   8. https://github.com/ZijunDeng/pytorch-semantic-segmentation [PyTorch]
   9. https://github.com/chainer/chainercv/tree/master/examples/segnet [Chainer]
   10. https://github.com/ykamikawa/keras-SegNet [Keras]

FCN – Fully Convolutional Network for Semantic Segmentation

FCNs são outro modelos de redes neurais convolucionais para segmnetação semântica.

Links:

1. Towards Data Science::Review: FCN (Semantic Segmentation)
2. [2015 CVPR] Fully Convolutional Networks for Semantic Segmentation
3. [2017 TPAMI] Fully Convolutional Networks for Semantic Segmentation
4. Site do Grupo de Pesquisa: http://fcn.berkeleyvision.org/
5. GitHub original de Evan Shelhamer: https://github.com/shelhamer/fcn.berkeleyvision.org
6. Link com vários modelos: Divam Gupta – Image Segmentation Keras : Implementation of Segnet, FCN, UNet and other models in Keras
7. Variantes dos Autores originais: Clockwork Convnets for Video Semantic Segmentation
8. Tutorial passo-a-passo usando VGG16 e FCN em TensorFlow puro: Medium::How to do Semantic Segmentation using Deep learning
9. Discussões e Tutoriais:
   1. Medium::Rediscovering Semantic Segmentation
10. Outras implementações:
    1. Tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation
    2. PyTorch Implementation of Fully Convolutional Networks (Training code to reproduce the original result is available)
    3. FCN for semantic segmentation – This is a simple FCN implementation in Pytorch. There are some differences from the original FCN
    4. https://github.com/vlfeat/matconvnet-fcn [MatConvNet]
    5. https://github.com/MarvinTeichmann/tensorflow-fcn [Tensorflow]
    6. https://github.com/aurora95/Keras-FCN [Keras]
    7. https://github.com/mzaradzki/neuralnets/tree/master/vgg_segmentation_keras [Keras]
    8. https://github.com/k3nt0w/FCN_via_keras [Keras]
    9. https://github.com/shekkizh/FCN.tensorflow [Tensorflow]
    10. https://github.com/seewalker/tf-pixelwise [Tensorflow]
    11. https://github.com/divamgupta/image-segmentation-keras [Keras]
    12. https://github.com/ZijunDeng/pytorch-semantic-segmentation [PyTorch]
    13. https://github.com/wkentaro/pytorch-fcn [PyTorch]
    14. https://github.com/wkentaro/fcn [Chainer]
    15. https://github.com/apache/incubator-mxnet/tree/master/example/fcn-xs [MxNet]
    16. https://github.com/muyang0320/tf-fcn [Tensorflow]
    17. https://github.com/ycszen/pytorch-seg [PyTorch]
    18. https://github.com/Kaixhin/FCN-semantic-segmentation [PyTorch]
    19. https://github.com/petrama/VGGSegmentation [Tensorflow]
    20. https://github.com/simonguist/testing-fcn-for-cityscapes [Caffe]
    21. https://github.com/hellochick/semantic-segmentation-tensorflow [Tensorflow]
    22. https://github.com/pierluigiferrari/fcn8s_tensorflow [Tensorflow]

Comparação de Enfoques de Segmentação Semântica

Siddhant Kumar traz uma comparação de iferentes enfoques de segmentação semântica. Está resumida na imagem abaixo:

Links de Comparações

1. Towards Data Science::Review: SegNet (Semantic Segmentation) – Differences from DeconvNet and U-Net

Links & Papers: Segmentação Semântica

Links

1. Awesome Semantic Segmentation
2. Qure AI: Guide to Semantic Segmentation with Deep Learning
3. Semantic-Segmentation – A list of all papers and resoureces on Semantic Segmentation
4. Really-awesome-semantic-segmentation – A list of all papers on Semantic Segmentation and the datasets they use
5. FCN – Full Code · GitHub – https://gist.github.com/khanhnamle1994/e2ff59ddca93c0205ac4e566d40b5e88
6. https://github.com/MarvinTeichmann/tensorflow-fcn/blob/master/fcn8_vgg.py
7. https://github.com/udacity/CarND-Semantic-Segmentation/
8. https://github.com/sagieppel/Fully-convolutional-neural-network-FCN-for-semantic-segmentation-Tensorflow-implementation
9. Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016
10. Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art [Webpage]

Papers

1. Shelhamer et.al. Fully Convolutional Networks for Semantic Segmentation (2016)
2. RTSeg: Real-time Semantic Segmentation Comparative Study
3. Indoor Scene Understanding in 2.5/3D: A Survey
4. A 2017 Guide to Semantic Segmentation with Deep Learning by Qure AI
5. A Review on Deep Learning Techniques Applied to Semantic Segmentation

Segmentação de Instâncias

Detecção de Objetos Integrada à Segmentação Semântica

1. Detectron – Detectron é o sistema da divisão de P&D em IA do Facebook que implementa algoritmos estado-da-arte para detecção de objetos, incluindo Mask R-CNN.
   A versão atual é escrita em Python e baseia no framework Caffe.
   1. https://github.com/facebookresearch/Detectron
   2. https://research.fb.com/downloads/detectron/
   3. Hackernoon: How to use Detectron — Facebook’s Free Platform for Object Detection
   4. Mask-R-CNN
      1. Mask R-CNN – Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick (2018) – a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition.
      2. Mask R-CNN for Object Detection and Segmentation – This is an implementation of Mask R-CNN on Python 3, Keras, and TensorFlow
      3. Keras implementation of MaskRNN instance aware segmentation as described in Mask R-CNN by Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick, using RetinaNet as base
2. Usando a Tensorflow Object Detection API:
   1. Towards Data Science – Using Tensorflow Object Detection to do Pixel Wise Classification
   2. Towards Data Science: Building a Custom Mask RCNN model with Tensorflow Object Detection
      1. https://github.com/priya-dwivedi/Deep-Learning/tree/master/Custom_Mask_RCNN

Integração de Princípios Tradicionais à Segmentação Semântica

Fazendo uma CNN melhorar Watershed

• Para células e imagens de microscópio: Towards Data Science: Instance segmentation: automatic nucleus detection
• Deep Watershed Transform for Instance Segmentation – Min Bai, Raquel Urtasun
• Uma implementação de Deep Watershed: https://github.com/min2209/dwt

Segmentação Panóptica

Este é um tópico bastante recente (2018!) e que está apenas começando a receber atenção de pesquisadores.

Links:

1. Panoptic Segmentation – Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár
2. Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network – Daan de Geus, Panagiotis Meletis, Gijs Dubbelman
3. Attention-guided Unified Network for Panoptic Segmentation – Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang (dezembro de 2018!)
4. GitHub::Awesome-Panoptic-Segmentation
5. GitHub::Topics-Panoptic-Segmentation
6. GitHub::COCO 2018 Panoptic Segmentation Task API (Beta version)

Como avaliar e validar a qualidade do seu modelo?

1. Reconhecimento de Padrões::Avaliando, Validando e Testando o seu Modelo: Metodologias de Avaliação de Performance

Aplicações no Browser

1. Medium::Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js
2. TensorFlow.js::BodyPix – Person Segmentation in the Browser
3. GitHub::BodyPix – Person Segmentation in the Browser

Datasets e Modelos Prontos

Datasets

1. ImageNet Tree View
2. COCO – Common Objects in Context
3. The PASCAL Visual Object Classes
4. Kitti Road dataset com download daqui
5. Photo Aesthetics Ranking Network with Attributes and Content Adaptation
6. AVA: A Large-Scale Database for Aesthetic Visual Analysis

Código

1. https://github.com/MarvinTeichmann/MultiNet
2. https://github.com/MarvinTeichmann/KittiSeg
3. https://github.com/vxy10/p5_VehicleDetection_Unet [Keras]
4. https://github.com/ndrplz/self-driving-car
5. https://github.com/mvirgo/MLND-Capstone
6. https://github.com/zhujun98/semantic_segmentation/tree/master/fcn8s_road

Copyright © 2018 Aldo von Wangenheim/INCoD/Universidade Federal de Santa Catarina